Scenario: I have multiple XA compliant databases fronted by different microservices which perform CRUD operations on them. I need to perform a 2 phase commit among these microservices. This means that I have a server running which makes API calls into these microservices to do some update, and these updates should be transactional.
We are planning to create a transaction manager to manage this.
Question: All the available solutions like Atomikos etc. mandate the different transactions to happen on the same server but in my case these happen in different microservices.
How can we provide transaction management in this case?
Ultimately we wanted to prepare transactions and then commit them in a different session as managed by our own transaction manager.
Is that possible?
It is definitely possible (you can do xa_prepare and xa_commit on separate sessions on most, if not all by now, resource managers) but indeed in the end you will end up writing essentially a Java EE (JTA) style transaction manager with transaction context propagation over REST or messaging or whatever communication mechanism you are using. This has been done eg in Rest-AT specification that Narayana/JBoss implemented and a few others.
Weblogic has an operator with a number of years on it now that takes it into the Kubernetes space and so XA/2PC can simply continue to be used there and Tuxedo will be putting out a product to achieve the same end (over Rest).
The saga pattern should also definitely be considered. It is not to be blindly accepted nor dismissed out of hand as a great pattern/fit in the microservices space. Use cases in transaction management, like any other area, continue to be more and more optimized and specialized and so the fact that it involves eventual consistency, compensation, etc. should not be non-starter in and of itself as it has a number of significant advantages as far as deployment models, scaling, and, to your point, the removal of XA distributed locks, etc. The best solution depends on the specific use case and it's requirements.
A number of the microservices frameworks, such as Narayana (WildFly/Quarkus/SpringBoot), Helidon, and even inside the Oracle DB itself, have Saga engines now. Full disclosure, I work at Oracle and am putting out a workshop on this very product in the next few weeks which will build on the existing "Simplifying Microservices with converged Oracle Database Workshop" which has a very basic choreography-based saga (as opposed to the orchestration-based products/engines I mentioned).
Very happy to talk more on this topic as I've spent the last 25 years writing transaction managers. :)
For the scenario described in the question, you should try using Oracle Transaction Manager for Microservices (MicroTx). It is a free product that comes with a transaction manager and client library for microservices written in Java and node.js. With this, you can create XA transactions involving multiple microservices.
Oracle MicroTx - https://www.oracle.com/database/transaction-manager-for-microservices
I don't think you mean a different Session exactly, but one Application tier level transaction where application components one different server are inside the transaction boundary.
The issue that you are facing is that those who created Microservices were not aware or experienced enough with information systems to understand these scenarios.
Microservices are essentially a false generalization and derivative of a stereotyping
Transactions and many other basic concepts that historically allow Enterprise information systems to exchange information globally without proprietary vendor locking are simply not part of the Microservices understanding.
So your question is really how do you retrofit the architecture concept to do normal day-to-day computer stuff.
In the end, if you keep solving those problems, you will be back to a Java EE application server. (Spring went through the same failings and just ended up wrapping and rebranding Java EE standard functions but with more obnoxious rhetoric)
My business logic on Glassfish can talk to the business login on WebLogic and the CICS tx on the mainframe, and everyone's databases and message queues all on different servers in one transaction. The XA spec lays out how to do this.
Related
I have a few microservices where each microservice has REST endpoints for CRUD operations.
I have to create a workflow that will start from one microservice with some initial input, but later outputs from a microservice can be used as input to other microservices. There can be some synchronous and asynchronous calls to these REST APIs.
I have looked for some of the workflow-engines but I do not think that I can create my workflow without writing any java code.
Should I write a separate microservice just for microservices orchestration? This orchestration microservice will know the exact workflow and can be configurable for inputs required to start the workflow, and it can also use some third-party workflow engines like Camunda to store the definition of the workflow.
Is this correct thinking to have a separate microservice just for microservices orchestration? Till now the existing microservices have no idea about the other microservices. There could be a chance that output from one microservice needs to be massaged before using as input for other microservice.
I have looked for some of the workflow-engines but I do not think that
I can create my workflow without writing any java code.
This depends on your business processes and the complexity of your workflow. Usually yes you will need to write some code to achieve it.
Should I write a separate microservice just for microservices
orchestration? This orchestration microservice will know the exact
workflow and can be configurable for inputs required to start the
workflow, and it can also use some third-party workflow engines like
Camunda to store the definition of the workflow.
Yes you can do that. I did something similar on a system using micro-services. This would be a very good Idea on the long run as you could configure your workflow based on environments as well. For example on your development machine you would have a little different workflow/configuration. This is practical for Developers or QA's testing their solutions. On the other hand on Staging/Production you can pre-define Customer setups/orchestration which you can reuse any time if you get new customers or users.
Is this correct thinking to have a separate microservice just for
microservices orchestration? Till now the existing microservices have
no idea about the other microservices. There could be a chance that
output from one microservice needs to be massaged before using as
input for other microservice.
Yes you can do that without problems although I would be careful with the name orchestration as this has another meaning in context in micro-service architecture(Docker, Docker-Swarm, Kubernetes). Similar examples would be some kind of EndToEndTest or Cross micro-service testing-micro-service. That would test cross micro-service business operations and assert the results. Usually business operations involve more then 1 micro-service so in order to test that you can use this approach. This micro-service would call APIs from multiple micro-services and test the results and scenarios based on your Business rules. Another example would be something like seeder-micro-service(which seems to be very similar to what you are trying to do here). This seeder-micro-service would be responsible for seeding(creating) test data to your micro-services. This test data is some basic setup/configuration data which you need in order to have your micro-service business processes to work. This is very handy for development machines or some test environments where you need to quickly setup an environment. Using this seeder-micro-service you can easily setup do your work or tests and dispose the environment(data) as you need it. This is especially useful for development machines setups but it can also be used on shared test environments and etc. Both of those examples are micro-services which server your needs and make your life easier to work with your system.
One final note regarding this:
Till now the existing microservices have no idea about the other
microservices.
They should be abstracted from each other in a way that they are not aware of internal implementation or data(separate databases) but they should communicate between each other in order to perform business operations which sometimes are cross micro-services. Like the typical example of payment-micro-service and order-micro-service from an online shop example. So it is fine that they know about each other and communicate but this communication has to be very carefully designed in order to avoid some common pitfalls.
They usually communicate with each other with direct calls over HTTP or some other protocol or through some message queue like Apache Kafka or RabbitMq or others. You can read more about it in this answer.
Yes, you should cover the orchestration part in a separate service. And, yes, go with a BPMN 2 process engine as orchestrator, as you already suspected. Yes, this may include writing a little code mostly for data mapping or connectors.
Benefits include for instance ootb support for:
state management and long running processes / persistence for data
versioning (!)
retries and error handling
tooling to modify state and date in case something went wrong
timeouts, parallel execution (if necessary)
scalability
graphical process model
audit trail
and end to end visibility in monitoring tools based on BPMN 2 model
ability to include business rules tasks (DMN) for more complex rules
combination of push and pull communication pattern and a/sync communication
business-IT alignment via BPMN 2
support for the various BPMN 2 events
standardization (skills, security, software quality, features)
...
This is a great related article about the WHY using an airline ticket booking as an example:
https://blog.bernd-ruecker.com/3-common-pitfalls-in-microservice-integration-and-how-to-avoid-them-3f27a442cd07
This is about the important design consideration in case you go with a process engine:
https://blog.bernd-ruecker.com/the-microservice-workflow-automation-cheat-sheet-fc0a80dc25aa
I faced a similar problem as in the original question. I'd a few microservices with simple dependencies that needed to be managed and did go down the path of writing my own microservice https://github.com/pedro-r-marques/workflow to manage the dependencies and do the orchestration. It uses a yaml definition file to describe the dependencies and rabbitmq for message passing. One could also replace the usage of rabbitmq by REST API calls front-ended by a load-balancer.
I have 3 micorservices one that serves request from UI and the other that serves request from public apis and the third which does some data processing and storing the data provided from the kafka topic by UI/public.
I have written common service and dao jar for the services, as the data is coming from the common data source.
If I dont have common service/dao then lot of code will be duplicated.
I am now feeling that this is causing coupling between the services.
Is it the right design?
Using a common DAO across microservices is right if it is making development faster and easier to understand for everyone, and wrong if it's not. You are right that this is creating some coupling between the services, but it's coupling that you could easily do away with if the DAOs for the services began to diverge. Since the final shared package will be inside each service's runtime, there would be zero issues introduced if one of the other services decided to stop using the DAO and use a different one.
That being said, you may have a larger coupling issue if all three services are using this DAO to connect to a shared database. If each is dependent on the same tables/schema, it makes it very hard for one service to diverge from the others and make independent schema changes without impacting the others.
We're working on a project , and we want to use some toggling feature tool like ff4j or togglz but we have a real constraints about performances, i mean we really need a tool with the less time of execution , i've checked a little bit ff4j and togglz but i don't know what is best for this solution, or may be if you know some other tools.
Context of project: its a netflix microservices architecture, so we have eureka,ribbon,zuul and microservices.
otherwise , if you have another solution , may be develop a sidecar please give me some ideas.
thank you in advance :)
Disclaimer : I created FF4j, as such I won't give you answer relative to performance comparison. I will provide architecture design principles.
Microservices means distributed architecture so you will have to store the state of your features in a common persistence storage (DB).
The cost of feature toggle framework won't be time to evaluate the feature state predicate (it is a simple condition) it will be the time to access the data from the persistence storage.
FF4j provides support for both REDIS and CONSUL:
Redis seems a good candidate as very fast for put/get and distribute.
Consul is also a good idea in distributed microservice : it provides a key-value store.
Eureka may does the same, I don't know, ff4j does not have store for it yet.
If you have to store your features in a slower DB such as SQL-Like then you might consider to use caching. FF4j provides some cacheProxy to handle such use cases.
Other Considerations :
Put the administration console only in a backend application not on each microservices (security + performance overhead)
Feature Toggle can do more with Configuration Management and monitoring.
You may want to have a look at this 15min talk exactly on that subject. LIVE DEMO starting at 7:10
and related github repository for sample with Spring-Cloud
Does anybody has an experience with Spring Integration project as embedded ESB?
I'm highly interesting in such use cases as:
Reading files from directory on schedule basis
Getting data from JDBC data source
Modularity and possibility to start/stop/redeploy module on the fly (e.g. one module can scan directory on schedule basis, another call query from jdbc data source etc.)
repeat/retry policy
UPDATE:
I found answers on all my questions except "Getting data from JDBC data source". Is it technically possible?
Remember, "ESB" is just a marketing term designed to sell more expensive software, it's not a magic bullet. You need to consider the specific jobs you need your software to do, and pick accordingly. If Spring Integration seems to fit the bill, I wouldn't be too concerned if it doesn't look much like an uber-expensive server installation.
The Spring Integration JDBC adapters are available in 2.0, and we just released GA last week. Here's the relevant section from the reference manual: http://static.springsource.org/spring-integration/docs/latest-ga/reference/htmlsingle/#jdbc
This link describes the FileSucker with Spring Integration. Read up on your Enterprise Integration patterns for more info I think.
I kinda think you need to do a bit more investigation your self, or do a couple of tries on some of your usecases. Then we can discuss whats good and bad
JDBC Adapters appear to be a work in progress.
Even if there is no specific adapter available, remember that Spring Integration is a thin wrapper around POJOs. You'll be able to access JDBC in any component e.g. your service activators.
See here for a solution based on a polling inbound channel adapter too.
What are the advantages and disadvantages of the Session Façade Core J2EE Pattern?
What are the assumptions behind it?
Are these assumptions valid in a particular environment?
Session Facade is a fantastic pattern - it is really a specific version of the Business Facade pattern. The idea is to tie up business functionality into discrete bundles - such as TransferMoney(), Withdraw(), Deposit()... So that your UI code is accessing things in terms of business operations instead of low level data access or other details that it shouldn't have to be concerned with.
Specifically with the Session Facade - you use a Session EJB to act as the business facade - which is nice cause then you can take advantage of all the J2EE services (authentication/authorization, transactions, etc)...
Hope that helps...
The main advantage of the Session Facade pattern is that you can divide up a J2EE application into logical groups by business functionality. A Session Facade will be called by a POJO from the UI (i.e. a Business Delegate), and have references to appropriate Data Access Objects. E.g. a PersonSessionFacade would be called by the PersonBusinessDelegate and then it could call the PersonDAO. The methods on the PersonSessionFacade will, at the very least, follow the CRUD pattern (Create, Retrieve, Update and Delete).
Typically, most Session Facades are implemented as stateless session EJBs. Or if you're in Spring land using AOP for transactions, you can create a service POJO that which can be all the join points for your transaction manager.
Another advantage of the SessionFacade pattern is that any J2EE developer with a modicum of experience will immediately understand you.
Disadvantages of the SessionFacade pattern: it assumes a specific enterprise architecture that is constrained by the limits of the J2EE 1.4 specification (see Rod Johnson's books for these criticisms). The most damaging disadvantage is that it is more complicated than necessary. In most enterprise web applications, you'll need a servlet container, and most of the stress in a web application will be at the tier that handles HttpRequests or database access. Consequently, it doesn't seem worthwhile to deploy the servlet container in a separate process space from the EJB container. I.e. remote calls to EJBs create more pain than gain.
Rod Johnson claims that the main reason you'd want to use a Session Facade is if you're doing container managed transactions - which aren't necessary with more modern frameworks (like Spring.)
He says that if you have business logic - put it in the POJO. (Which I agree with - I think its a more object-oriented approach - rather than implementing a session EJB.)
http://forum.springframework.org/showthread.php?t=18155
Happy to hear contrasting arguments.
It seems that whenever you talk about anything J2EE related - there are always a whole bunch of assumptions behind the scenes - which people assume one way or the other - which then leads to confusion. (I probably could have made the question clearer too.)
Assuming (a) we want to use container managed transactions in a strict sense through the EJB specification then
Session facades are a good idea - because they abstract away the low-level database transactions to be able to provide higher level application transaction management.
Assuming (b) that you mean the general architectural concept of the session façade - then
Decoupling services and consumers and providing a friendly interface over the top of this is a good idea. Computer science has solved lots of problems by 'adding an additional layer of indirection'.
Rod Johnson writes "SLSBs with remote interfaces provide a very good solution for distributed applications built over RMI. However, this is a minority requirement. Experience has shown that we don't want to use distributed architecture unless forced to by requirements. We can still service remote clients if necessary by implementing a remoting façade on top of a good co-located object model." (Johnson, R "J2EE Development without EJB" p119.)
Assuming (c) that you consider the EJB specification (and in particular the session façade component) to be a blight on the landscape of good design then:
Rod Johnson writes
"In general, there are not many reasons you would use a local SLSB at all in a Spring application, as Spring provides more capable declarative transaction management than EJB, and CMT is normally the main motivation for using local SLSBs. So you might not need th EJB layer at all. " http://forum.springframework.org/showthread.php?t=18155
In an environment where performance and scalability of the web server are the primary concerns - and cost is an issue - then the session facade architecture looks less attractive - it can be simpler to talk directly to the datbase (although this is more about tiering.)