Is there a way to deploy a FastAPI application so that memory state cannot be persisted between requests? The goal is to avoid leaking any data between requests in a multi-tenant application.
Starting up the application from scratch for every request seems not feasible since it takes too long. Is there a way in which the application is launched for every instance of the service but individual requests are handled by workers or threads that get purged after the request is handled so that any static property, singleton instance and such is destroyed and the next request is handled with clean memory?
FastAPI is basically stateless by default. It actually takes extra work to persist data across requests through methods such connection pooling, reading a value from Redis, and so on. If you consider things such as starting up the server, loading a configuration, setting up path redirects, and so on to be "state", then FastAPI will not work for your purposes.
When you say "memory state", it sounds like you are trying to partition off instances of FastAPI server from each other so that they do not even use the same memory. This is not going to be a viable solution because most web servers, FastAPI included, are not designed for this type of segregating. By default, the requests from one tenant will not have anything to do with the requests from another tenant unless you write additional code that allows them to become related; so separating the concerns of the different tenants becomes a matter for the programmer, not the server's memory.
Instead, if you absolutely cannot let requests from multiple tenants inhabit the same memory, you'd be better off giving different tenants their own subdomain on the DNS level. Spin up a VPS and instance of your FastAPI program for each of them. That will truly prevent the requests from one tenant share any memory or state with the others.
Does Spring Session management take care of asynchronous calls?
Say that we have multiple controllers and each one is reading/writing different session attributes. Will there be a concurrency issue as the session object is entirely written/read to/from external servers and not the attributes alone?
We are facing such an issue that the attributes set from a controller are not present in the next read... this is an intermittent issue depending on the execution of other controllers in parallel.
When we use the session object from the container we never faced this issue... assuming that it is a direct attribute set/get happening right on to the session object in the memory.
The general use case for the session is storing some user specific data. If I am understanding your context correctly, your issue describes the scenario in which a user, while for example being authenticated from two devices (for example a PC and a phone - hence withing the bounds of the same session) is hitting your backend with requests so fast you face concurrency issues around reading and writing the session data.
This is not a common (and IMHO reasonable) scenario for the session, so projects such as spring-data-redis or spring-data-gemfire won't support it out of the box.
The good news is that spring-session was built with flexibility in mind, so you could of course achieve what you want. You could implement your own version of SessionRepository and manually synchronize (for example via Redis distributed locks) the relevant methods. But, before doing that, check your design and make sure you are using session for the right data storage job.
This question is very similar in nature to your last question. And, you should read my answer to that question before reading my response/comments here.
The previous answer (and insight) posted by the anonymous user is fairly accurate.
Anytime you have a highly concurrent (Web) application/environment where many different, simultaneous HTTP requests are coming in, accessing the same HTTP session, there is always a possibility for lost updates caused by race conditions between competing concurrent HTTP requests. This is due to the very nature of a Servlet container (e.g. Apache Tomcat, or Eclipse Jetty) since each HTTP request is processed by, and in, a separate Thread.
Not only does the HTTP session object provided by the Servlet container need to be Thread-safe, but so too do all the application domain objects that your Web application puts into the HTTP session. So, be mindful of this.
In addition, most HTTP session implementations, such as Apache Tomcat's, or even Spring Session's session implementations backed by different session management providers (e.g. Spring Session Data Redis, or Spring Session Data GemFire) make extensive use of "deltas" to send only the changes (or differences) to the Session state, there by minimizing the chance of lost updates due to race conditions.
For instance, if the HTTP session currently has an attribute key/value of 1/A and HTTP request 1 (processed by Thread 1) reads the HTTP session (with only 1/A) and adds an attribute 2/B, while another concurrent HTTP request 2 (processed by Thread 2) reads the same HTTP session, by session ID (seeing the same initial session state with 1/A), and now wants to add 3/C, then as Web application developers, we expect the end result and HTTP session state to be, after request 1 & 2 in Threads 1 & 2 complete, to include attributes: [1/A, 2/B, 3/C].
However, if 2 (or even more) competing HTTP requests are both modifying say HTTP sessoin attribute 1/A and HTTP request/Thread 1 wants to set the attribute to 1/B and the competing HTTP request/Thread 2 wants to set the same attribute to 1/C then who wins?
Well, it turns out, last 1 wins, or rather, the last Thread to write the HTTP session state wins and the result could either be 1/B or 1/C, which is indeterminate and subject to the vagaries of scheduling, network latency, load, etc, etc. In fact, it is nearly impossible to reason which one will happen, much less always happen.
While our anonymous user provided some context with, say, a user using multiple devices (a Web browser and perhaps a mobile device... smart phone or tablet) concurrently, reproducing this sort of error with a single user, even multiple users would not be impossible, but very improbable.
But, if we think about this in a production context, where you might have, say, several hundred Web application instances, spread across multiple physical machines, or VMs, or container, etc, load balanced by some network load balancer/appliance, and then throw in the fact that many Web applications today are "single page apps", highly sophisticated non-dumb (no longer thin) but thick clients with JavaScript and AJAX calls, then we begin the understand that this scenario is much more likely, especially in a highly loaded Web application; think Amazon or Facebook. Not only many concurrent users, but many concurrent requests by a single user given all the dynamic, asynchronous calls that a Web application can make.
Still, as our anonymous user pointed out, this does not excuse the Web application developer from responsibly designing and coding our Web application.
In general, I would say the HTTP session should only be used to track very minimal (i.e. in quantity) and necessary information to maintain a good user experience and preserve the proper interaction between the user and the application as the user transitions through different parts or phases of the Web app, like tracking preferences or items (in a shopping cart). In general, the HTTP session should not be used to store "transactional" data. To due so is to get yourself into trouble. The HTTP session should be primarily a read heavy data structure (rather than write heavy), particularly because the HTTP session can be and most likely will be accessed from multiple Threads.
Of course, different backing data stores (like Redis, and even GemFire) provide locking mechanisms. GemFire even provides cache level transactions, which is very heavy and arguable not appropriate when processing Web interactions managed in and by an HTTP session object (not to be confused with transactions). Even locking is going to introduce serious contention and latency to the application.
Anyway, all of this is to say that you very much need to be conscious of the interactions and data access patterns, otherwise you will find yourself in hot water, so be careful, always!
Food for thought!
I have a Spring-boot based microservice which is currently being hit from a Mobile-APP. Now we are developing a browser base client for the same microservice. Request & Response parameters between mobile-App and browser are same. Number of users in mobile-APP is around 10000 per second and for browser is around 20000 per second. Hence, there would be more than 30000 hits to this microservice each second.
We know that "Spring controllers are singletons (there is just one instance of each controller per web application) ".
Will it be a good approach (with respect of performance) to have two separate Controllers for this same microservice, one for mobile-App users and other for browser users ? Will it improve microservice performance by having two instances running in parallel ?
I am looking the best way to handle increasing number of hits through both the channels ?
Any suggestions would be highly appreciated.
When You have the same request and response, retrieved to browser and mobile clients there is no point in creating two diff controllers or services. Keep your app simple with one controller to do the job. With this your service just sees the mobile and web client in same way.
Whenever there is increase in load that has to be handled by the app, you can go for horizontal scaling, using a routing, load balancer service like zuul, nginx.
Just scale up/down the instances behind the load balancer according to the load you need to handle.
I am using a proprietary IoC mechanism in my asp.net mvc3 application (on IIS7) that saves state in [ThreadStatic] fields and therefore relies on an assumption that HttpApplication.BeginRequest, HttpApplication.EndRequest and the whole synchronous execution of the (single) request they relate to are executed on the same thread.
Is that assumption correct?
Is that assumption correct?
No, this assumption is not correct and there's evidence for it. The only reliable per request storage mechanism in ASP.NET is HttpContext.Items.
Never use [ThreadStatic] fields to store per-request values in an ASP.NET application. For example if you have an asynchronous controller you could very well have the engine draw one request from the thread pool to begin serving the request, then initiate an asynchronous operation relying on an IOCP (I/O Completion Port) and finally draw another thread from the pool to finish the request. So you could have 2 different threads serving the same HTTP request.
Absolutely never rely on the fact that the HTTP request will be served by the same thread.
This could be true in some cases for synchronous requests but remember that this is just an implementation detail. This could change without any notice from one version of .NET to another. You should never rely on it and never use [ThreadStatic] in ASP.NET. This could bite you very badly.
I find that WCF service will take 8-10 seconds to load the first hit. After that it will take less than a second.
Any thoughts?
Probably due to .NET's cold start. Have you looked at setting up the IIS Warmup Module which initializes dependancies before an initial request?
From the Learn IIS website
Decrease the response time for first requests by pre-loading worker processes. The IIS Application Warm-Up module lets you configure the Web application to be pre-loaded before the first request arrives so that the worker process responds to the first Web request more quickly.
Increase reliability by pre-loading worker processes when overlapped recycling occurs. Because the recycled worker process in an overlapped recycling scenario only communicates its readiness and starts accepting requests after it finishes loading and initializing the resources as specified by the configuration, pre-loading the dependencies reduces the response times for the first requests.
Customize the pre-loading of applications. You can configure the IIS Application Warm-Up module to initialize Web applications by using specific Web pages and user identities. This makes it possible to create specific initialization processes that can be executed synchronously or asynchronously, depending on the initialization logic. In addition, these procedures can use specific identities to ensure a proper initialization.