Looking at eShopOnContainers, the microservice reference architecture from Microsoft. I see that for each service, in Program.cs a call is made to host.MigrateDbContext. This, in turn, executes all of the EF migrations for the given context.
In a real-world orchestrator isn't is possible that numerous containers for the same service could be spun up almost simultaneously? And if that happened, isn't it likely that multiple containers trying to execute the same migrations would deadlock or cause other issues?
Is this something that wasn't dealt with because it is beyond the scope of a reference project or does EF have something built in to handle concurrency that I'm not seeing?
I've found that there are numerous approaches to this problem, each with their own strengths and weaknesses. Some are straightforward... bringing the entire app down, updating schema, and then bringing the app back online. Some implement the schema changes as a series of smaller changes, each of which are both forward and backward compatible, allowing zero downtime. Still others leverage built-in or third-party tools written specifically to address this task.
So, to answer my own question, this topic was almost certainly omitted because it was beyond the scope of the eShopOnContainers project/eBooks. The right choice for you will vary based on your project's size, complexity, acceptable downtime, etc.
Related
I realize the benefits of workflow engine such as easy to understand communication, easy waiting, parallelism and compensative actions with informative graphical model. The concept is great and more manageable than dogmatic event driven architecture without central coordinator and specified flow.
We are currently using legacy workflow engine to orchestrate microservices in insurance business. Over the time chunks of business logic and little helper scripts has creeped into process model, which is not developer friendly solution to maintain and test with continuous integration standards. Also the lack of available expertise and future support is a huge risk from the project management perspective.
I played around with Camunda and Activiti, but immediately faced compability issues with Spring Boot 3 and a lack of up to date examples and general knowledge outside of relatively small user community. This gives me a bad feeling of drowning into the same swamp as we are now in the future.
We planned design our own Java based orchestrator, which just invokes specified microservices in a specified order when the process is started or user task is completed. The orchestrator will also handle monitoring and versioning of the process flow. It's up to microservices to validate their business context and halt the process by raising user tasks if necessary. When user task is completed, the orchestrator restarts the whole process from the beginning with all tasks cleared. It is the responsibility of microservices to no-op when their work is already done in the previous run. Eventually, the process will reach it's end and finish. This solution would be a good balance of modern DX and coordinated process management.
Is there examples or name for such an idempotent orchestrated architecture?
You only get into the challenge of aligning dependencies between your services and the process engine (and other components) if you tightly couple the orchestration / engine with the services. Happened to me many times in the past, too. If you separate the engine (called remote process engine with Camunda 7, only architecture with Camunda 8), then you are not influenced by its dependencies. Try for instance the Camunda RUN distribution and the external task pattern or C8 SaaS to get to a cleaner, decoupled architecture. See Bernd Ruecker's reasoning here.
Details will depend on your specific requirements, but I would definitely advise anyone against building a homegrown solution. There are enough options in the market and these times are over. Requirements grow over time. There are security vulnerabilities to be aware of and to fix, etc. High maintenance, no market for resources, no synergies, you would need to maintain proprietary knowledge in the company and cannot achieve the same level of quality and feature richness as a more broadly used solution can. For a list of options see for instance Bernd Ruecker's articles. Among the available options I would personally prefer an orchestrator, which uses a graphical process modelling approach based on the BPMN 2 standand. It helps clarity, knowledge transfer, and Business-IT alignment and the standard is a vendor-independent skill set.
There is no need to build your own. Use temporal.io open source project. Besides Java SDK it supports Go, Typescript/Javascript, Python, PHP.
The project started at Uber in 2016. There are hundreds of companies using it for mission critical applications.
Setup
My app uses gRPC for frontend/backend separation
Backend (server + services) is written in Python
Goals
I'd love to achieve live-updates including:
Updating existing services
Adding new services
Removing services
Problem
I can achieve #1 by using Python's importlib feature. But there are fewer options for adding/removing services. It seems to depend on the app's gRPC implementation. The major constraints seem to be the fact that servicers can only be registered before running the server, i.e., through the call to
add_MyServiceServicer_to_server()
So does adding reflection support, which is through the call to
service_names = [
MyService_pb.DESCRIPTOR.services_by_name[''].full_name,
...
]
reflection.enable_server_reflection(service_names, my_server)
Solution Candidates
Approach 1: Have one servicer per service, much like the official SayHello example of gRPC
Approach 2: Have one giant servicer that includes all the other services as its RPC methods.
Approach 1 seems to be intuitive, but it won't support adding/removing services while the server is running.
Approach 2 seems promising, but it is confusing by sticking the entire universe in a single servicer. And I'm not sure how gRPC's thread pool would like this approach.
Questions
Can I achieve my goals with these approaches?
If true, which one is better in terms of both maintainability and performance?
If false, are there alternatives?
As per #DougFawley's comments,
Typically microservices would have multiple replicas deployed, and be restarted in a rolling update when new services are added.
"What's the best user experience like in this scenario?" ->
microservice clients should expect and be resilient to RPC failures.
They can happen for many other reasons in steady state. Typically you
will run multiple replicas and when one is restarted, if you
gracefully shut it down, clients will have a chance to create
connections to other backends and no RPCs will fail. But if they do
fail, clients should retry and will use another backend. Users should
not really be impacted by this.
In short, it's a bad idea to add/remove services without rebooting server. So for this reason, I'd better adopt the one-to-one servicer-service binding and not hack it for this particular live-update intent.
I was going through design principles but could not understand this principle(avoid Micro layers), what would the significance be. I tried to google it but could not find any examples or explanation for this design principle. Could it possible for someone to explain this with example,what advantages it has in which scenarios? Does layering not localizes changes and reduces ripple effect of changes in software?
You’ve misinterpreted the way the principle is written. The author wasn’t trying to say “avoid micro services”. They were trying to say “When dealing with a micro service, don’t keep adding features or functionality to it. Instead, add an additional micro service to deliver the new functionality.”
The intent is to help you keep each micro service focused on a single task. This simplifies any system that depends on your service. And, it means you can more easily update your service — possibly by quickly rewriting it if you come up with a better performing design, for example. It’s hard to say “we’re going to rewrite our server” if that’s a six month task. It’s much easier when it’s only a one- or two-sprint task.
This thread needs a reboot coz of 2 reasons.
The Clean Code book that I have doesn't have a mention of Micro Layers. So not sure where the now "omni"-downloaded Clean Code cheat sheet got this from.
It would help if someone can guide me to where in the Clean Code book I can read on this one.
Am not fully satisfied that we re discussing Micro Layers in the scope of Micro Services. Bringing in an arch pattern Micro Services is not helping discuss a topic in a book that was written at a base level of Code and OOAD and a bit of design.
Instead for practical illustration purpose a code level example of the above statement is needed.
I’ve been doing a lot of googling regarding managing dependencies between microservices. We’re trying to move away from big monolithic app into micro-services in order to scale organizationally and be able to develop faster and with multiple teams working in parallel.
However, as we’re trying to functionally partition the monolith into the microservices, we see how intertwined business logic and data really is. This was not a problem when we were sitting on top of one big DB and were able to do big relational joins. But with microservices, this becomes a problem.
One solution is to make microservice-A go to 5-10 other microservices to get necessary data (this is equivalent of DB view with join). Another solution is to make microservice-A listen to events from 5-10 other services and populate local storage with relevant into (this is an equivalent of materialized view). Either way, microservice-A is coupled with 5-10 other services, and if new info is needed in microservice-A, the some of the services that it depends upon might will need to be release prior to microservice-A. Please note that microservice-A is itself depended upon by other services. Bottom line, we end up with DISTRIBUTED dependency hell.
Many articles advocate for second solution – i.e. something along the lines of Event Sourcing, Choreography, etc.
I would appreciate any shared experiences, recommendations and insights.
Philometor.
While not technically an "answer", I can definitely share some of my observations and experiences. Your question concerning services calling other services for database operations reminded me of a project where an architect sold senior management on the idea of "decoupling" persistence from the rest of the applications by implementing hundreds of REST interfaces in what essentially was a distributed DAO pattern in front of a very large enterprise database. The project ended up exactly the way I predicted - a dismal failure.
Microservices aren't about turning a monolithic application into a distributed monolithic application. In my example project above, the monolith was turned into a stove-piped, fragile, chaotic mess, with the coupling only moved to service contracts instead of Java class method signatures, and with a performance hit so bad the application was unusable. Last I heard they are still running their original monolith.
Microservices should be more of a vertical partitioning of your application and not a horizontal one. In my opinion it's better to think in terms of business function partitioning rather than "converting" an existing monolith. There's no rule that determines how big a microservice must be, but it should be big enough to do one complete synchronous function without needing to directly depend on outside services (as much as possible) to complete its work. If a microservice performs a complex business function that affects 50 tables, so be it! It owns those many tables. Ideally if a service goes down, it should affect only that business functionality it's responsible for, and not directly affect other services. As you can see, this thinking is the complete opposite from that which produced the distributed mess in my project example.
Not only do you need to ensure that the motivation behind replacing monoliths with microservices is sound, but also you need to step outside the monolith and revisit the actual business and begin partitioning that instead. Like everything else, baby steps are the way to go. Start with one small complete business function, and convert that into a single microservice instead of trying to replace a monolith all at once.
Currently, I deal with microservices on a daily basis at my 9-5. Most everything that I touch is written in PHP, and as only a software engineer, SysOps manages everything that has to do with apps running, etc. I have a little familiarity in how the infrastructure and build pipeline is setup, but I still am not a SysOps or DevOps guy.
With that said, I love Golang and for a side project, I am creating a fairly large web application with a lot of moving parts. Writing and designing the code is easy as I have learned a lot from my day job, but deploying and managing Golang web apps (as they are executables) is quite different than updating files for apache to serve.
I have researched a lot on how I would build and deploy my microservice apps, but I keep on thinking of more problems that will need to be solved along the way. I have tinkered with the idea of using Docker for all of this, but I would rather not have the added complexity of learning that and managing storage for all of the images as that could be large.
Is there a best practice or a good way to manage Golang applications after they have been deployed? I would need a way to keep track of all the microservice processes to be able to see if they are still up and to be able to stop them when a new build is going to be deployed.
As for the setup, just assume that all the microservices will be run on the server, not in a container or in a VM. They will all need to be managed, but also able to act upon independently. Jenkins will be used for building and deploying. I will be using Consul for service discovery and possibly configuration, and most likely health checks on the services. I'm thinking of having each microservice register itself to consul when started and deregister when stopping.
Again, I am looking for a solution that is hopefully not just "Docker". I also had thoughts into creating a deploy service that manages the services (add and remove), as well as registering them in Consul. So if I cannot find a better solution, I might go that path. Any help is appreciated.
** Sorry if my question was confusing, but since a couple people answered on the wrong topic at hand, I will try to clarify. I don't need any help making the microservices, or even know anything about them. I brought that point up as to why I need to ask my question. Basically what I need is just the ability to manage the running go processes of all my microservices so I can do deployments and be able to stop and start processes to update the code. It is easier when you have to worry about one app, but when you can have up to 10-15 difference microservices they become harder to keep track of. After my own research, it seems that Supervisord is what I am looking for, but I'm not sure. That is the direction I am going in with this question. Thanks.
Golang is great to use for microservices, but I would say there is not so big difference of managing golang or other languages microservices.
What I would say is golang specific:
you don't need to install anything on servers since golang is compiled to single library
you can take advantage of std lib golang rpc package and gob binary decoding, instead of usin 3rd party solution (gorpc, protocol buffer etc)
Other than that you need to use your own judgement. There is plenty of ways of doing one things in microservices world; one day you will implement solution A but when after 3 month you will see that its better to do B, do that.
In internet, there is so much reading about microservices. I will recommend you 2 good resurces: https://books.google.co.uk/books/about/Building_Microservices.html?id=RDl4BgAAQBAJ&source=kp_cover&redir_esc=y&hl=en
And article: http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
Remember, microservices are not a golden bullet, they often can help making application easier to maintain and grow, but from the other side require lot of additional work, consequence in specifying API contracts and strong devops culture.