I have a simple rest endpoint that executes Postgres procedure.
This procedure returns the current state of device.
For example:
20 devices.
Client app connect to API and make 20 responses to that endpoint every second.
For x clients there are x*20 requests.
For 2 clients 40 requests.
It causes a big cpu load on Postgres server only if there are many clients and/or many devices.
I didn’t create it but I need to redesign it.
How to limit concurrent queries to db only for it? It would be a hot fix.
My second idea is to create background worker that executes queries only one in the same time. Then the endpoint fetches data from memory.
I would try the simple way first. Try to reduce
the amount of database connections in the pool OR
the amount of working threads in the build-in Tomcat.
More flexible option would be to put the logic behind a thread pool limiting the amount of working threads. It is not trivial, if the Spring context and database is used inside a worker. Take a look on a Spring annotation #Async.
Offtopic: The solution we are discussing here looks like a workaround. The discussed solution alone will most probably increase the throughput only by factor 2 maybe 3. It is not JEE conform and it will be most probably not very stable. It is better to refactor the application avoiding such a problem. Another option would be to buy a new database server.
Update: JEE compliant solution would be to implement some sort of bulkhead pattern. It will limit the amount of concurrent running requests and reject it, if the some critical number is reached. The server application answers with "503 Service Unavailable". The client application catches this status and retries a second later (see "exponential backoff").
Related
I have a Spring boot app that use HikariCP for Postgres connection pooling.
Recently I've set up tracing to collect some data how time is spent when handling a request to a specific endpoint.
My assumptions are that when using HikariCP:
The first connection to the database while handling the request might be a bit slower
Subsequent connections to the database should be fast (< 10 ms)
However, as the trace shows, the first connection is fast (< 10 ms). And while some subsequent connections during the same request handling are also fast (< 10 ms), I frequently see some subsequent connections taking 50-100ms, which seems quite slow to me, although I'm not sure if this is to be expected or not.
Is there anything I can configure to improve this behavior?
Maybe good to know:
The backend in question doesn't really see any other traffic right now, so it's only handling traffic when I manually send requests to it
I've changed maximumPoolSize to 1 to rule out that the issue is that it uses different connections in the context of 1 request and that's what causes the issue. The same behavior is still seen.
I use the default Hikari settings, I don't change them.
I do think something is wrong with your pool configuration or your usage of the pool if it takes roughly 10 ms to get an already initialized connection from your pool. I would expect it to be sub-millisecond... Are you sure you are using the pool correctly?
Make sure you are using as new versions of pool and driver as possible, and make sure that connectionTestQuery is not set, as that would execute a query every time the connection is obtained from the pool. The defaults should be good enough for the rest of the settings.
Debug logs could be one thing help figure out what is happening, metrics on the pool another. Have a look at Spring Boot Actuator, it will help you with that...
To answer your actual question on how you can improve the situation given it actually takes roughly 10 ms to obtain a connection: Do not obtain and return the connection to the pool for every query... If you do not want to pass the connection around in your code, and if it suits your use case, you can make this happen easily by making sure your whole request is wrapped in a transaction. See the Spring guide on managing transactions.
I was doing a project that needs to support a cluster of 30k nodes, all those nodes periodic calls the api to get data.
I want to have the maximum amount of concurrent get operation per second, and due to it is get operation, it must be in synced way.
And my local pc is 32GB 8Core, spring boot version is 2.6.6, configurations are like
server.tomcat.max-connections=10000
server.tomcat.threads.max=800
I use jmeter to do concurrent test, and the through out is around 1k/s, average response time is 2 seconds.
Is there any way to make it support more requests per second?
Hard to say without details on the web service, implementation of what it actually does and where the bottleneck actually is (threads, connections, CPU, memory or others) but, as a general recommendation, using non-blocking APIs would help but it should then be full non-blocking to actually make a real difference.
I mean that just adding Webflux and have blocking DB would not improve so much.
Furthermore, all improvements in execute time would help so check if you can improve the code and maybe have a look at trying to go native (which will come "built in" in Boot 3.X btw)
I have a spring boot webflux application which by default uses netty.
One of the business requirements that we have mandates that requests should time out within 2 seconds.
When very few requests are sent to the app, everything is fine but when the request load is increased (Like over 40 or 50 concurrent per second by Jmeter) sometimes all of them time out due to each taking longer than the 2-second threshold.
I have spent a long time reading things online and looking into what could be causing this issue but with no success. When requests are sent concurrently most end up taking a long time and the problematic part is where an external HTTTP request is made to other microservice. All my tests are local and I have tested the microservices and they seem fast enough to handle a big load so the microservices themselves are not the issue.
I know that netty uses event loop and does not create a thread per request.
I believe there are likely synchronous tasks that are blocking those few netty threads. For this reason I have done massive refactoring and have ".publishOn(Schedulers.boundedElastic())" or ".subscribeOn(Schedulers.boundedElastic())" in the Mono reactive chains. After the refactoring Most of the operations seem to be running on elastic threads and not the "reactor-http-nio-x" (According to the logs) but doing so has not helped the main issue and the problem still exists.
It will be a huge help if someone could direct me to what I should be doing. At this point, I have no more improvements to make, and think I might have been looking at this the wrong way and my approach has not been correct.
I have not attached any code sine the application is big and I do not still know where the actual problem lies.
I've encountered the same problem. I've didn't find the root cause of this, but when I switched from WebClient to RestTemplate with dedicated thread pool per client (external service) then the problem was solved. I've run a blockhound to find if I block somewhere in the stream, but it didn't find anything. I've also tried deploying my application with increased number of NIO worker thread pool (by default it's equal to cores number) and there was some improvement, but after all RestTemplate yielded the best performance. So I'm still on Webflux stack, but I don't use WebClient anymore and the performance on high load is fine.
I'm working on a web application frontend to a legacy system which involves a lot of CPU bound background processing. The application is also stateful on the server side and the domain objects needs to be held in memory across the entire session as the user operates on it via the web based interface. Think of it as something like a web UI front end to photoshop where each filter can take 20-30 seconds to execute on the server side, so the app still has to interact with the user in real time while they wait.
The main problem is that each instance of the server can only support around 4-8 instances of each "workspace" at once and I need to support a few hundreds of concurrent users at once. I'm going to be building this on Amazon EC2 to make use of the auto scaling functionality. So to summarize, the system is:
A web application frontend to a legacy backend system
task performed are CPU bound
Stateful, most calls will be some sort of RPC, the user will make multiple actions that interact with the stateful objects held in server side memory
Most tasks are semi-realtime, where they have to execute for 20-30 seconds and return the results to the user in the same session
Use amazon aws auto scaling
I'm wondering what is the best way to make a system like this distributed.
Obviously I will need a web server to interact with the browser and then send the cpu-bound tasks from the web server to a bunch of dedicated servers that does the background processing. The question is how to best hook up the 2 tiers together for my specific neeeds.
I've been looking at message Queue systems such as rabbitMQ but these seems to be geared towards one time task where any worker node can simply grab a job form a queue, execute it and forget the state. My needs are a little different since there could be multiple 'tasks' that needs to be 'sticky', for example if step 1 is started in node 1 then step 2 for the same workspace has to go to the same worker process.
Another problem I see is that most worker queue systems seems to be geared towards background tasks that can be processed anytime rather than a system that has to provide user feedback that I'm dealing with.
My question is, is there an off the shelf solution for something like this that will allow me to easily build a system that can scale? Would love to hear your thoughts.
RabbitMQ is has an RPC tutorial. I haven't used this pattern in particular but I am running RabbitMQ on a couple of nodes and it can handle hundreds of connections and millions of messages. With a little work in monitoring you can detect when there is more work to do then you have consumers for. Messages can also timeout so queues won't backup too greatly. To scale out capacity you can create multiple RabbitMQ nodes/clusters. You could have multiple rounds of RPC so that after the first response you include the information required to get second message to the correct destination.
0MQ has this as a basic pattern which will fanout work as needed. I've only played with this but it is simpler to code and possibly simpler to maintain (as it doesn't need a broker, devices can provide one though). This may not handle stickiness by default but it should be possible to write your own routing layer to handle it.
Don't discount HTTP for this as well. When you want request/reply, a strict throughput per backend node, and something that scales well, HTTP is well supported. With AWS you can use their ELB easily in front of an autoscaling group to provide the routing from frontend to backend. ELB supports sticky sessions as well.
I'm a big fan of RabbitMQ but if this is the whole scope then HTTP would work nicely and have fewer moving parts in AWS than the other solutions.
I have a .NET 3.5 BasicHttpBinding no security WCF service hosted on IIS 6.0.
I have service throttling bumped up as per MS recommendations.
My service operation is getting called a few hundreds of time conccurrently, and at some point the client gets a timeout exception (59:00, that's whats set in the server and client timeouts).
If I raise the timeout it just hits the new limit.
It seems like the application just "freezes" somewhere and we have not been able to figure out how this happens.
WCF tracing on the server side doesn't come up with anything.
Any ideas regarding what could be the issue?
Thanks
I assume your WebService is not using the new async/await especially wrt the database calls. In that case its because you are blocking your limited threads.
In more detail. IIS/ASP.net only creates a limited number of threads to handle requests. The first...say 8 requests spin up threads and start working. At some point they will hit the database (I am assuming a traditional n-tier app). Those threads sleep. The next say...992 requests hit IIS and are held in a queue.
At some point the database calls return, process stuff...send data to the client. Another 8 requests are dequeued...hit the database...etc...
However each set of 8 requests takes a finite time to complete. With over 900 requests ahead of them, the last 100 or so threads will take at the very least 100 * latency * number of roundtrips before they can start up. If your latency * number of roundtrips is high...your last request will take a long time before it even gets dequeued, hence the timeout.
Two remedies exists. The first, create more threads....will use up all your memory and your IIS crashes. The second is to use .net 4.5 and async/await.
See here for more information