Imagine your company has an event-driven complex architecture managing all sorts of operations: payments, processes and so on and so forth.
The code running in said step-functions is actually pretty easy to understand, mainly consists in API calls to various services.
Complexity is given by the business logic behind these operations and what I want to do is breaking it down having something that will facilitate future onboardings.
How would you imagine a documentation written to explain product/business processes rather than technical ones?
I have been thinking about using Swagger to expose our event-bus and have something to start from. I would then write the docs using Gitbook, but I am opened to using other tools. Any help is highly appreciated!
Related
I am thinking what is the best way to structure your micro-services, in the past the team I was working with used Axon Framework and PostgreSQL and each microservice had its own event store in the PostgreSQL database, then we built communication between using REST.
I am thinking that it would be smarter to have all microservices talk to the same event store as we would be able to share events faster instead of rewriting the communication lines using REST.
The questions that follows from the backstory is:
What is the best practice for having an event store
Would each service have its own? Would they share the same eventstore?
Where would I find information to inspire and gather more answers? As searching the internet for best practices and how to structure the Event Store seems like searching for a needle in a haystack.
Bear in mind, the question stated is in no way aimed at Axon Framework, but more the general idea on building scalable and good code. As the applications would work with each own event store for write model and read models.
Thank you for reading and I wish you all the best
-- Me
I'd add a slightly different notion to Tore's response, although the mainline is identical to what I'm sharing here. So, I don't aim to overrule Tore, just hoping to provide additional insight.
If the (micro)services belong to the same Bounded Context, then they're allowed to "learn about each other's language."
This language thus includes the events these applications publish and store.
Whenever there's communication required between different Bounded Contexts, you'd separate the stores, as one context shouldn't be bothered by the specifics of another context.
Hence it is beneficial to deduce what services belong to which Bounded Context since that would dictate the required separation.
Axon aims to support this by allowing multiple contexts with the Axon Server, as you can read here.
It simply allows the registration of applications to specific contexts, within which it will completely separate all message streams (so commands, events, and queries) and the Event Store.
You can also set this up from scratch yourself, of course. Tore's recommendation of Kafka is what's used quite broadly for Event Streaming needs between applications. Honestly, any broadcast type of infrastructure suits event distribution, as that's how events are typically propagated.
You want to have one EventStore per service, just as you would want to have one relation database per service for a non EventSourced system.
Sharing a database/eventstore between services creates coupling and we have all learned the hard way that this is an anti-pattern today.
If you want to use a event log to share events across services, then Kafka is a popular choice.
Important to remember that you only do event-sourcing within a service bounded context.
For example,
You have an IT estate where a mix of batch and real-time data sources exists from multiple systems, e.g. ERP, Project management, asset, website, monitoring etc.
The aim is to integrate the datasources into a cloud environment (agnostic).
There is a need for reporting and analytics on combinations of all data sources.
Inevitably, some source systems are not capable of streaming, hence batch loading is required.
Potential use-cases for performing functionality/changes/updates based on the ingested data.
Given a steer for creating a future-proofed platform, architecturally, how would you look to design it?
It's a very open-end question, but there are some good principles you can adopt to help direct you in the right direction:
Avoid point-to-point integration, and get everything going through a few common points - ideally one. Using an API Gateway can be a good place to start, the big players (Azure, AWS, GCP) all have their own options, plus there's lots of decent independent ones like Tyk or Kong.
Batches and event-streams are totally different, but even then you can still potentially route them all through the gateway so that you get the centralised observability (reporting, analytics, alerting, etc).
Use standards-based API specifications where possible. A good REST based API, based off a proper resource model is a non-trivial undertaking, not sure if it fits with what you are doing if you are dealing with lots of disparate legacy integration. If you are going to adopt REST, use OpenAPI to specify the API's. Using this standard not only makes it easier for consumers, but also helps you with better tooling as many design, build and test tools support OpenAPI. There's also AsyncAPI for event/async API's
Do some architecture. Moving sh*t to cloud doesn't remove the sh*t - it just moves it to the cloud. Don't recreate old problems in a new place.
Work out the logical components in your new solution: what does each of them do (what's it's reason to exist)? Don't forget ancillary components like API catalogues, etc.
Think about layering the integration (usually depending on how they will be consumed and what role they need to play, e.g. system interface, orchestration, experience APIs, etc).
Want to handle data in a consistent way regardless of source (your 'agnostic' comment)? You'll need to think through how data is ingested and processed. This might lead you into more data / ETL centric considerations rather than integration ones.
Co-design. Is the integration mainly data coming in or going out? Is the integration with 3rd parties or strictly internal?
If you are designing for external / 3rd party consumers then a co-design process is advised, since you're essentially designing the API for them.
If the API's are for internal use, consider designing them for external use so that when/if you decide to do that later it's not so hard.
Taker a step back:
Continually ask yourselves "what problem are we trying to solve?". Usually, a technology initiate is successful if there's a well understood reason for doing it, which has solid buy-in from the business (non-IT).
Who wants the reporting, and why - what problem are they trying to solve?
As you mentioned its an IT estate aka enterprise level solution mix of batch and real time so first you have to identify what is end goal of this migration. You can think of refactoring applications. If you are trying to make it event driven then assess the refactoring efforts and cost. Separation of responsibility is the key factor for refactoring and migration.
If you are thinking about future proofing your solution then consider Cloud for storing and processing your data. Not necessary it will be cheap but mix of Cloud and on-prem could be a way. There are services available by cloud providers to move your data in minimal cost. Cloud native solutions are there for performing analysis on your data. Database migration service in AWS or Azure can move data and then capture on-going changes. So you can keep using on-prem db & apps and perform analysis for reporting on cloud. It will ease out load on your transactional DB. Most data sync from on-prem to cloud is near real time.
I was going through design principles but could not understand this principle(avoid Micro layers), what would the significance be. I tried to google it but could not find any examples or explanation for this design principle. Could it possible for someone to explain this with example,what advantages it has in which scenarios? Does layering not localizes changes and reduces ripple effect of changes in software?
You’ve misinterpreted the way the principle is written. The author wasn’t trying to say “avoid micro services”. They were trying to say “When dealing with a micro service, don’t keep adding features or functionality to it. Instead, add an additional micro service to deliver the new functionality.”
The intent is to help you keep each micro service focused on a single task. This simplifies any system that depends on your service. And, it means you can more easily update your service — possibly by quickly rewriting it if you come up with a better performing design, for example. It’s hard to say “we’re going to rewrite our server” if that’s a six month task. It’s much easier when it’s only a one- or two-sprint task.
This thread needs a reboot coz of 2 reasons.
The Clean Code book that I have doesn't have a mention of Micro Layers. So not sure where the now "omni"-downloaded Clean Code cheat sheet got this from.
It would help if someone can guide me to where in the Clean Code book I can read on this one.
Am not fully satisfied that we re discussing Micro Layers in the scope of Micro Services. Bringing in an arch pattern Micro Services is not helping discuss a topic in a book that was written at a base level of Code and OOAD and a bit of design.
Instead for practical illustration purpose a code level example of the above statement is needed.
I have a requirement to persist some data in a table (single table). The data is coming from UI. Do i need to write just the system API and persist the data OR i need to write process and system API both? I don't see a use of process API in this case. Please suggest. Is it always necessary to access system API through process API or system API can be invoked without process API as well.
I would recommend a fine-grained approach to this. We should be following it through the experience layer even though we do not have must customization to the data.
In short, an experience layer API and directly calling System layer API (if there is no orchestration/data conversion/formatting needed)
Why we need a system API & experience API? A couple of points.
System API should be more attached to the underlying system. And if
in case, in the future, it changes then it should not impact any of
the clients.
Secondly, giving an upper layer gives us the feasibility to add
different SLAs, policies, logging and lots more, to different
clients. Even if you have a single client right now it's better to
architect for the future. Reusing is the key advantage of these APIs.
Please check Pattern 2 in this document
That is a question for the enterprise architect in your organisation. In this case, the process API would probably be a simple proxy for the system API, but that might not always be the case in future. Also, it is sometimes useful to follow a standard architectural pattern even if it creates some spurious complexity in the implementation. As always, there are design trade-offs and the answer will depend on factors that cannot be known by people outside of your organisation.
I have a question about microservice implementation. right now I am using an api gateway to process all get request to my individual services and using kafka to handle asynchronous post put and delete request. Is this a good way of handling of handling request in a microservice architecture?
Your question is too unspecific to give a good answer. What is a good architecture totally depends on the details of your use cases. Are you serving web pages, streaming media, amass data for analysis, or something completely different? We would also need to know what are you requirements in terms of concurrency, consistency and scalability? What are the constraints for budget/size of development teams, ease of development, dev skills, etc?
For example the decisions you have taken may be considered good if you have strong requirements for a highly scalable input of large data sets and very frequent data collection as well as the team to support it. But it may be considered bad if you have a small team only and are trying to get a quick and cheap MVP for a new service that has limited scalability requirements (because the complexity of the solution slows down your development unnecessarily).
It may be good because the development team is familiar with those technologies and can effectively develop with those. Or it may be bad because your team does not know anything about those and the investment in learning those will not be justifiable by long term gains.
Don't forget that one of the ideas of the microservices architectural style is that each service can be owned by a distinct team that makes its own decisions about what technology to use for implementation (for whatever reason: ease of development, business reasons etc). So in other words the microservices style embraces the old wisdom architecture follows organization.
Here a link to a recommended further read.