According to my research, I found out that in thread per request model, every request that comes spawns a new thread. Let's say I had 100 requests, I'd be having 100 threads running at once. Coming to event looped model (similar to spring webflux), we have a main thread that listens to the requests and delegates tasks to other threads.
Now let's say we have 100 requests on event looped model. Here, the main thread will be free to listen but it will also have threads which will be waiting for response from DB or network, just like thread per request model. How does it make event looped model more scalable.
The key difference between Tomcat with Servlet API < 3.1 and servers as Netty powered with a Spring WebFlux is the way which IO and requests are processed : blocking or non-blocking.
Spring WebFlux favors the second approach :
Part of the answer is the need for a non-blocking web stack to handle
concurrency with a small number of threads and scale with fewer
hardware resources.
So to sum that, by using the Spring WebFlux API, much less threads will be created for as many as client http requests because a thread is not dedicated to a single client http request in this model.
The no-blocking approach means that : whatever the time to process a request, a thread that handles that will not block the application and keep the thread waited for a long time but will process another request during this time.
Take that example : your Rest or Mvc controller receives a request and the essential of the task to perform is requesting the database. With a blocking approach you create one thread by http request. With a no blocking approach, the thread delegates to the database, may serve other requests and that thread or another of the pool will go on the processing when the interaction with the database is finished.
Related
I'm designing a system where an HTTP service with multiple threads accepts request to perform work. These requests are placed into a multiprocessing queue and sent downstream to a worker process where the work is performed (let's assume that we have a reasonable expectation that work can be handled quickly and the HTTP threads aren't blocking for a long time)
The issue that I can't figure out how to solve is - once the worker process is done processing the request, how are the results returned to the specific producer that produced a request?
I considered having another multiprocessing queue - a "results" queue - that each producer has a handle to, and they can wait on this queue for the results. The issue is that there's no guarantee that a specific producer will pull the results for their request from this queue, it might go to some other producer, and that other producer won't hold the open connection to the requesting HTTP client so it won't be able to do anything with the results.
I've included a simple system diagram below that shows the producer threads and worker process
One solution here would be to have the worker process write the results to some data store, e.g. Redis, under a random key created by the producer, and the producer could watch this key for the result. However, I would prefer to avoid adding an external storage system if possible as the overhead of serializing to/from Redis would be non-trivial in some cases.
I don't think the language should matter here, but in case it does, this would be developed in Python using some standard microframework (like FastAPI)
EDIT - I thought of one possible solution. I can have another thread in the producer process that is responsible for reading from a "response" multiprocessing queue from the worker process. All other producer threads can then query some thread-safe data structure within this "response reader" thread for their specific results (which will be placed under some unique key generated by the producer)
The main issue I'm struggling with now is how to scale this to multiple producer processes (each with multiple producer threads) and multiple worker processes that are distinct (worker A handles different jobs from worker B)
Suppose an ASP.NET WebAPI request arrives at a controller method.
Suppose the request represents an 'event' which needs processed. The event has multiple operations associated with it that should be performed in parallel. For example, each operation may need to call out to a particular REST endpoint on other servers, which are I/O bound operations that should get started as soon as possible and should not wait for one to return before starting the next one.
What is the most correct/performant way to implement this pattern?
I've read that using Task.Run is a bad idea, because it just grabs additional ThreadPool threads, leaving the main request thread idle/blocked. While that makes sense if I was running a single task, I'm not sure that advice applies in this case.
For example, if the event has 4 operations that needed completed (each having possibly multiple I/O bound calls of their own), I would call Task.Run in a loop 4 times to initialize each operation, then wait on the resulting tasks with Task.WaitAll.
Q1: Would the main request thread be returned to the ThreadPool for use by another request while waiting for Task.WaitAll to return, or would it just hog the main thread leaving it idle until Task.WaitAll completes?
Q2: If it hogs the main thread, could that be resolved by marking the controller method with the async keyword, and using an await Task.WhenAll call instead? I'd imaging that this would return the main thread to the pool while waiting, allowing it to be used for other requests or event operations.
Q3: Since Task.Run queues up a work item that could be blocked on an I/O bound call, would performance improve if the operations were all implemented with async and used await calls on Task-based asynchronous I/O methods?
Regarding the whole approach of using Task.Run for the event's operations, the goal is just get all of the operation's I/O bound calls started as soon as possible. I suppose if (as in Q3) all operations were async methods, I could just get them all started on the main request thread in a loop, but I'm not sure that would be better than starting them with separate Task.Run calls. Maybe there's a completely different approach that I'm unaware of.
So I'm using Play as my MVC framework. My web application simply does calls to a Postgres database to feed data to the views. I'm currently using AJAX in the view to get the data. I figured there are times when the database will lag a bit in sending the data so I would use AJAX to allow other elements in the view to load.
Now my question is, since I'm already using AJAX in the view, should I be using Promises in the controllers? Would it make a difference if I used Promises? I haven't had enough experience to figure out how having asynchronous actions both in the view and in the controller can affect my web application. My intuition says that the AJAX action is enough for a web application with about 100 - 150 hits a day.
What are your insights on this?
The point of non-blocking I/O and Promises is not about speed or latency, but about better use of resources. By default, Play gives you one thread per CPU core. This works very well when all of the work you're doing on those threads is extremely fast - that is, you avoid expensive computations and use only non-blocking I/O. However, all JDBC calls are synchronous, so they will block the few available threads for a long time, and if you have enough traffic, any new requests will have to queue up, increasing load time for your pages.
Therefore, whether you should use Promises - or, more accurately, a separate thread pool - for DB calls has nothing to do with whether you make those DB calls on the initial page load or via AJAX calls. It's simply about how much traffic you expect, how many threads you have, and how long you'll be using each thread. For most applications, the best practice is to run DB calls on a separate thread pool so they don't block the main worker threads.
For example, configure a thread pool (ie, ExecutionContext) in application.conf as follows:
akka {
actor {
db-context {
fork-join-executor {
parallelism-factor = 20.0
parallelism-max = 200
}
}
}
}
Then use that thread pool in your DB code:
def dbLookup(someId: Int): Future[SomeDbValue] = {
val dbExecutionContext = Akka.system.dispatchers.lookup("db-context")
Future {
// DB code to fetch SomeDbValue
}(dbExecutionContext)
}
See Play Framework Thread Pools for full instructions and Play Framework: async I/O without the thread pool and callback hell for more background info.
(This probably duplicates the question ASP.NET MVC4 Async controller - Why to use?, but about webapi, and I do not agree with answers in there)
Suppose I have a long running SQL request. Its data should be than serialized to JSON and sent to browser (as a response for xhr request). Sample code:
public class DataController : ApiController
{
public Task<Data> Get()
{
return LoadDataAsync(); // Load data asynchronously?
}
}
What actually happens when I do $.getJson('api/data', ...) (see this poster http://www.asp.net/posters/web-api/ASP.NET-Web-API-Poster.pdf):
[IIS] Request is accepted by IIS.
[IIS] IIS waits for one thread [THREAD] from the managed pool (http://msdn.microsoft.com/en-us/library/0ka9477y(v=vs.110).aspx) and starts work in it.
[THREAD] Webapi Creates new DataController object in that thread, and other classes.
[THREAD] Uses task-parallel lib to start a sql-query in [THREAD2]
[THREAD] goes back to managed pool, ready for other processing
[THREAD2] works with sql driver, reads data as it ready and invokes [THREAD3] to reply for xhr request
[THREAD3] sends response.
Please, feel free to correct me, if there's something wrong.
In the question above, they say, the point and profit is, that [THREAD2] is not from The Managed Pool, however MSDN article (link above) says that
By default, parallel library types like Task and Task<TResult> use thread pool threads to run tasks.
So I make a conclusion, that all THREE THREADS are from managed pool.
Furthermore, if I used synchronous method, I would still keep my server responsive, using only one thread (from the precious thread pool).
So, what's the actual point of swapping from 1 thread to 3 threads? Why not just maximize threads in thread pool?
Are there any clearly useful ways of using async controllers?
I think the key misunderstanding is around how async tasks work. I have an async intro on my blog that may help.
In particular, a Task returned by an async method does not run any code. Rather, it is just a convenient way to notify callers of the result of that method. The MSDN docs you quoted only apply to tasks that actually run code, e.g., Task.Run.
BTW, the poster you referenced has nothing to do with threads. Here's what happens in an async database request (slightly simplified):
Request is accepted by IIS and passed to ASP.NET.
ASP.NET takes one of its thread pool threads and assigns it to that request.
WebApi creates DataController etc.
The controller action starts an asynchronous SQL query.
The request thread returns to the thread pool. There are now no threads processing the request.
When the result arrives from the SQL server, a thread pool thread reads the response.
That thread pool thread notifies the request that it is ready to continue processing.
Since ASP.NET knows that no other threads are handling that request, it just assigns that same thread the request so it can finish it off directly.
If you want some proof-of-concept code, I have an old Gist that artificially restricts the ASP.NET thread pool to the number of cores (which is its minimum setting) and then does N+1 synchronous and asynchronous requests. That code just does a delay for a second instead of contacting a SQL server, but the general principle is the same.
The profit of asynchronous actions is that while the controller is waiting for the sql query to finish no threads are allocated for this request, while if you used a synchronous method a thread would be locked up in the execution of this method from the start to end of that method. While SQL server is doing its job the thread isn't doing anything but waiting.
If you use asynchronous methods this same thread can respond to other requests while SQL server is doing its thing.
I believe your steps are wrong at step 4, I don't think it will create a new thread to do the SQL query. At 6 there isn't created a new thread, it is just one of the available threads that will be used to continue from where the first thread left off. The thread at 6 could be the same as started the async operation.
The point of async is not to make an application multi threaded but rather to let a single threaded application carry on with something different instead of waiting for a response from an external call that is executing on a different thread or process.
Consider a desk top application that shows stock prices from different exchanges. The application needs to make a couple of REST / http calls to get some data from each of the remote stock exchange servers.
A single threaded application would make the first call, wait doing nothing until it got the first set of prices, update it's window, then make a call to the next external stock price server, again wait doing nothing until it got the prices, update it's window... etc..
We could go all multi threaded kick off the requests in parallel and update the screen in parallel, but since most of the time is spent waiting for a response from the remote server this seems overkill.
It might be better for the thread to:
Make a request for the first server but instead of waiting for the answer leave a marker, a place to come back to when the prices arrive and move on to issuing the second request, again leaving a marker of a place to come back to...etc.
When all the requests have been issued the application execution thread can go on to dealing with user input or what ever is required.
Now when a response from one of the servers is received the thread can be directed to continue from the marker it laid down previously and update the window.
All of the above could have been coded long hand, single threaded, but was so horrendous going multi threaded was often easier. Now the process of leaving the marker and coming back is done by the compiler when we write async/await. All single threaded.
There are two key points here:
1) Multi threaded does still happen! The processing of our request for stock prices happens on a different thread (on a different machine). If we were doing db access the same would be true. In examples where the wait is for a timer, the timer runs on a different thread. Our application though is single threaded, the execution point just jumps around (in a controlled manner) while external threads execute
2) We lose the benefit of async execution as soon as the application requires an async operation to complete. Consider an application showing the price of coffee from two exchanges, the application could initiate the requests and update it's windows asynchronously on a single thread, but now if the application also calculated the difference in price between two exchanges it would have to wait for the async calls to complete. This is forced on us because an async method (such as one we might write to call an exchange for a stock price) does not return the stock price but a Task, which can be thought of as a way to get back to the marker that was laid down so the function can complete and return the stock price.
This means that every function that calls an async function needs to be async or to wait for the "other thread/process/machine" call at the bottom of the call stack to complete, and if we are waiting for the bottom call to complete well why bother with async at all?
When writing a web api, IIS or other host is the desktop application, we write our controller methods async so that the host can execute other methods on our thread to service other requests while our code is waiting for a response from work on a different thread/process/machine.
I my opinion the following describes a clear advantage of async controllers over synchronous ones.
A web application using synchronous methods to service high latency
calls where the thread pool grows to the .NET 4.5 default maximum of
5, 000 threads would consume approximately 5 GB more memory than an
application able the service the same requests using asynchronous
methods and only 50 threads. When you’re doing asynchronous work,
you’re not always using a thread. For example, when you make an
asynchronous web service request, ASP.NET will not be using any
threads between the async method call and the await. Using the thread
pool to service requests with high latency can lead to a large memory
footprint and poor utilization of the server hardware.
from Using Asynchronous Methods in ASP.NET MVC 4
if there is a synchronized block in the service() method of say for example in ActionServlet of struts, how will multiple requests/threads work if it is a busy site having large number of hits.
Will each thread wait for the next other one to release lock from the synchronized block? Will that create delay in response
Don't synchronize the service method of a servlet.
If you synchronize the service method of a servlet, you are in fact making a "reservation on access" for a thread at a time for that instance of the Servlet.
The Struts ActionServlet class is a HttpServlet and basically the doGet and doPost methods are of interest here. If we are to speak about Struts, the process method is the main entry point, but the same principle applies to all methods as it does for the general service method.
The idea is this.
When you declare a servlet in your web.app, the servlet container (e.g. Tomcat) will create only one instance of that servlet. This means there is only one instance to serve all requests.
If more requests arrive at the same time, each request thread gets it chance at the service method because there is no synchronization enforced.
If you have 10 request threads, each will execute concurrently in the service method. This is normally safe because processing done in the service method does not involve any state related to the current request it is handling. You go into issues if you add state to your servlets. Here is an article with more details on the subject.
Now back to Struts.
Struts uses a pattern called a Front Controller with the ActionServlet being that controller. This will in turn delegate specific requests to specific Action classes as specified in its configuration (a.k.a struts-config.xml).
All incoming request pass though here. If you place synchronization at this point (the Struts process method or the servlet service method higher up) you are reserving the servlet for a thread at a time. In case of struts, you are reserving all processing of a request to a single thread at a time.
That means that if 10 request arrive simultaneousely, in the case without synchronization all can execute side by side, while in the case with synchronization request 2 will have to wait until request 1 is done, 3 waits for 2 and so on (i.e. requests are sequentially processed). And this means poor performance.
Maybe for the lucky user that made request 1 there will be no performance impact but number 10 will have to wait. Then how about number 100? 200?
There is no need to synchronize the entry point if you program your application with thread safety in mind. If your application is of the sort that you just can't avoid synchronization, then synchronizing the entry point will reduce performance.
P.S. Just one other thing. If you are thinking into moving the synchronization lower in the process, namely the Action classes, note that they are not thread safe either and there is only one instance of it inside the Struts framework.