I had an interesting discussion with a friend recently and we were talking about a client server architecture system where the server side was actually many servers (for load balancing). On a single server, I can see how thread safety can be achieved when many clients try to update the state of the system but how is this achieved in a multi-server load balanced environment? Having a single server acting as a gate to the multi-server load balancing farm that has the responsibility for thread safety seems like it would create a choke point and negate the load balancing features.
Can anyone explain how this would be achieved?
TIA
There are different techniques to ensure thread safety across multiple serves. One relies on the Thread Pool Pattern another one on the virtualization of server farms.
Related
Setup
My app uses gRPC for frontend/backend separation
Backend (server + services) is written in Python
Goals
I'd love to achieve live-updates including:
Updating existing services
Adding new services
Removing services
Problem
I can achieve #1 by using Python's importlib feature. But there are fewer options for adding/removing services. It seems to depend on the app's gRPC implementation. The major constraints seem to be the fact that servicers can only be registered before running the server, i.e., through the call to
add_MyServiceServicer_to_server()
So does adding reflection support, which is through the call to
service_names = [
MyService_pb.DESCRIPTOR.services_by_name[''].full_name,
...
]
reflection.enable_server_reflection(service_names, my_server)
Solution Candidates
Approach 1: Have one servicer per service, much like the official SayHello example of gRPC
Approach 2: Have one giant servicer that includes all the other services as its RPC methods.
Approach 1 seems to be intuitive, but it won't support adding/removing services while the server is running.
Approach 2 seems promising, but it is confusing by sticking the entire universe in a single servicer. And I'm not sure how gRPC's thread pool would like this approach.
Questions
Can I achieve my goals with these approaches?
If true, which one is better in terms of both maintainability and performance?
If false, are there alternatives?
As per #DougFawley's comments,
Typically microservices would have multiple replicas deployed, and be restarted in a rolling update when new services are added.
"What's the best user experience like in this scenario?" ->
microservice clients should expect and be resilient to RPC failures.
They can happen for many other reasons in steady state. Typically you
will run multiple replicas and when one is restarted, if you
gracefully shut it down, clients will have a chance to create
connections to other backends and no RPCs will fail. But if they do
fail, clients should retry and will use another backend. Users should
not really be impacted by this.
In short, it's a bad idea to add/remove services without rebooting server. So for this reason, I'd better adopt the one-to-one servicer-service binding and not hack it for this particular live-update intent.
Can micro-service interact with downstream service through localhost origin, since my all service is running in same server is that is correct approach ? I found that interacting a downstream service with domain name takes much time when compared to localhost. i was curious to know whether we can do like this ?
You're right, you can communicate with other services running in the same host with localhost. It's completely fine and when thinking about network round trips, it's beneficial.
But,
what if you want to scale the services?
What if you want to move any of the services to a different host?
While considering at least these scenarios, binding to a specific host is not worth. And this is applicable if you are using the IP of the host.
*I found that interacting a downstream service with domain name takes much time when compared to localhost.*.
I see what you're saying.
Microservices architecture is not a silver bullet for software development design and always come with tradeoffs
And about your deployment strategy Multiple Service Instances per Host pattern.
How you are going to handle if your services have different resource requirements?
say what if one of your services is utilizing all the host resource?
What if you need to scale out one independent service?
How you are going to ensure the availabilities of your services?
..
..
So there are so many questions you must consider before going with a pattern in microservices. It all depends on your requirements.
If your services are on the same server you should using a message broker or mechanism like grcp to talk between your services so doesn't matter if your orgin is. If you are using HTTP to communicate between your micro services then it totally not gain any advantages of micro services architecture and your architecture is flawed.
Microservice is a concept, it does not force you to where you deploy your application and how they might call each other. You may deploy your microservices on different virtual machines that are hosted on the same physical server. The whole point is you need to have a reason for everything that you decide to do with your architecture.
The first question is why you have split your application into different microservices? for only carrying the word of microservice on your architecture or having better control on the business logic, scalability, and maintainability of the project?
These are important things you need to take care of them when you are designing an application. draw the big picture of your product, how it's going to be used. which service/component is mostly being used by the customers, does keeping it with other microservices on the same server makes performance issues or not? what if any issue happens to the server and whole applications would be unreachable.
In my work place, we have a java system (tomcat,spring,hibernate,soap+rest web services). Some of the web services require the server to save state. For example, while performing some long service, the client cannot call the same service again, as long as it is not finished.
Currently, we don't support clustering, in order to avoid running the above mentioned long service, we use locks or synchronize blocks. In order to support clustering we consider using Hazel Cast (share the locks across the instances), will it work?
Is this the right solution?
Yes, maybe as the most typical use case; you can use distributed locks or other distribured data structures to share state between different servers (or jvms).
I'm about to build a new system and I want maximum availability! I'll have to use Windows!
I will have clients talking to my system using webservices. I'll also get data from surrounding systems. This data is delivered using messaging, MQ-series and MSMQ.
The system will produce some data that is sent back to the surrounding systems using queues.
After new data has come to the system different processes will use this data to do diffrent tasks, like printing, writing to databases etc.
To achieve high availablity I'm planning to have two versions of the system running in parallel on two different machines. The clients will try to use the first server thats responds correctly.
I think an ideal soultion would be that the incomming data from anyone of the two servers is placed in a COMMON queue(on a third machine?). Data in the queue can be picked up by processes on both servers(think producer-consumer pattern).
I think that maybe NServiceBus will suits my needs. I have a few questions according to the above.
Can a queue be shared between two servers? I dont want data to be stuck on a server if its gets down. I that case I want the other server to keep processing.
Can two(or more) "consumers"/processes on different machines pick data from a common queue?
Any advice is welcome!
The purpose of NSB distributor is not to address availability issues but to address scale issues, distributors help scaling out systems at a low cost.
By looking at the description, your system consist of WebService endpoits, multiple databases and queuing infrastructure. If you want to achieve complete high-availability you will have to make sure there are no single points of failures. In order to do that you will need,
A load balanced web farm for web service endpoints (2 or more servers)
Application cluster for queues and applications that relies on those queues.
Highly available database server, again clustered.
On top of everything a good SAN.
But if you are referring to being available to consumers, you just have to make sure target queues and webservice endpoints are available. And making sure the overall architecture promotes deferred execution.
Two or more applications can read a MSMQ queue remotely but thats something you don't want to do since it's based on DTC. And that's a real performance killer.
Some references
[http://blogs.msdn.com/b/clustering/archive/2012/05/01/10299698.aspx][1]
[http://msdn.microsoft.com/en-us/library/ms190202.aspx][2]
In short you will want to use the distributor... http://support.nservicebus.com/customer/portal/articles/859556-load-balancing-with-the-distributor
The key thing here is that the distributor node is a single point of failure so you want to run it on a cluster.
I'm porting a huge application to Windows Azure. It will have a web service frontend and a processing backend. So far I thought I would use web roles for servicing client requests and worker roles for backend processing.
Managing two kinds of roles seems problematic - I'll need to decide how to scale two kinds of roles and also I'll need several (at least two) instances of each to ensure reasonable fault tolerance and this will slightly increase operational costs. Also in my application client requests are rather lightweight and backend processing is heavyweight, so I'd expect that backend processing would consume far more processing power than servicing client requests.
This is why I'm thinking of using web roles for everything - just spawn threads and do both servicing requests and backend processing in each instance. This will make the role more complicated but will I guess simplify management. I'll have more instances of a uniform role and better fault tolerance.
Is it a good idea to reuse web roles for backend processing? What drawbacks should I expect?
Sounds like you already have a pretty good idea of what to think about when using multiple roles:
Cost for 2 instances to meet SLA (although some background tasks really don't need SLA if the end user doesn't see the impact)
Separate scale units
However: If you run everything in one role, then everything scales together. If, say, you have an administrative web site on port 8000, you might have difficulty reaching it if your user base is slamming the main site on port 80 with traffic.
I blogged about combining web and worker roles, here, which goes into a bit more detail along what we're discussing here. Also, as of some time in March, the restriction of 5 endpoints per role was lifted - see my blog post here for just how far you can push endpoints now. Having this less-restrictive endpoint model really opens up new possibilities for single-role deployments.
From what I understand your are asking if it makes sense to consolidate service layers so that you only have to deal with a single layer. At a high level, I think that makes sense. The simpler the better, as long as it's not so simple that you can't meet your primary objectives.
If your primary objective is performance, and the calls to your services are inline (meaning that the caller is waiting for an answer), then consolidating the layers may help you in achieving greater performance because you won't have to deal with the overhead of additional network latency of additional physical layers. You can use the Task Parallel Library (TPL) to implement your threading logic.
If your primary objective is scalability, and the calls to your services are out-of-band (meaning that the caller implements a fire-and-forget pattern), then using processing queues and worker roles may make more sense. One of the tenets of cloud computing is loosely coupled services. While you have more maintenance work, you also have more flexibility to grow your layers independendly. Your worker roles could also use the TPL mentioned above so that you can deploy your worker roles on larger VMs (say with 4CPUs, or 8), which would keep the number of instances deployed to a minimum.
My 2 cents. :)
I would suggest you to develop them as separated roles: a web role and a worker role, and then just combine them into a single web role.
this way, in the future you can easaly convert to real separated roles, if needed.
for more details:
http://www.31a2ba2a-b718-11dc-8314-0800200c9a66.com/2010/12/how-to-combine-worker-and-web-role-in.html
http://wely-lau.net/2011/02/25/combining-web-and-worker-role-by-utilizing-worker-role-concept/