I was doing a project that needs to support a cluster of 30k nodes, all those nodes periodic calls the api to get data.
I want to have the maximum amount of concurrent get operation per second, and due to it is get operation, it must be in synced way.
And my local pc is 32GB 8Core, spring boot version is 2.6.6, configurations are like
server.tomcat.max-connections=10000
server.tomcat.threads.max=800
I use jmeter to do concurrent test, and the through out is around 1k/s, average response time is 2 seconds.
Is there any way to make it support more requests per second?
Hard to say without details on the web service, implementation of what it actually does and where the bottleneck actually is (threads, connections, CPU, memory or others) but, as a general recommendation, using non-blocking APIs would help but it should then be full non-blocking to actually make a real difference.
I mean that just adding Webflux and have blocking DB would not improve so much.
Furthermore, all improvements in execute time would help so check if you can improve the code and maybe have a look at trying to go native (which will come "built in" in Boot 3.X btw)
Related
I have a simple rest endpoint that executes Postgres procedure.
This procedure returns the current state of device.
For example:
20 devices.
Client app connect to API and make 20 responses to that endpoint every second.
For x clients there are x*20 requests.
For 2 clients 40 requests.
It causes a big cpu load on Postgres server only if there are many clients and/or many devices.
I didn’t create it but I need to redesign it.
How to limit concurrent queries to db only for it? It would be a hot fix.
My second idea is to create background worker that executes queries only one in the same time. Then the endpoint fetches data from memory.
I would try the simple way first. Try to reduce
the amount of database connections in the pool OR
the amount of working threads in the build-in Tomcat.
More flexible option would be to put the logic behind a thread pool limiting the amount of working threads. It is not trivial, if the Spring context and database is used inside a worker. Take a look on a Spring annotation #Async.
Offtopic: The solution we are discussing here looks like a workaround. The discussed solution alone will most probably increase the throughput only by factor 2 maybe 3. It is not JEE conform and it will be most probably not very stable. It is better to refactor the application avoiding such a problem. Another option would be to buy a new database server.
Update: JEE compliant solution would be to implement some sort of bulkhead pattern. It will limit the amount of concurrent running requests and reject it, if the some critical number is reached. The server application answers with "503 Service Unavailable". The client application catches this status and retries a second later (see "exponential backoff").
I wrote a sample WebFlux application, which read just some data from redis and do small cpu job(md5 calculation), repeats several time.
It use spring data redis reactive
'org.springframework.boot:spring-boot-starter-data-redis-reactive'
Codes that connects redis is like this.
reactiveStringRedisTemplate.opsForValue().get(keyname)
redis server runs at localhost
redis-server &
Full codes , you can find here
https://github.com/mouse500/redperf
It's a simple codes
And for test, I call API (/wredis) with Jmeter to loadtest
Problem that I think is...
this application doesn't reach maximum TPS,
It reaches CPU around 40% on my local PC.
Even though it has more room of CPU, it doesn't work any harder
Why doesn't it use resource fully?
In case of some other method to connect redis,
(I put connection proxy ,written in nodejs)
It showed much much higher CPU and got much more TPS.
So I don't think it is not about Redis server performance.
Problem seems issue at "calling Redis from WebFlux application with lettuce"
How can I make this sample application show a maximum TPS ( CPU reach 100%) ?
What option can I try?
When I look into Jetty. I see this sentence and as a newbie for Jetty.
Load generators should be written in asynchronous programming style, so that limited threads does not limit the maximum number of users that can be simulated. If the generator is not asynchronous, then a thread pool of 2000 may only be able to simulate 500 or less users. The Jetty HttpClient is an ideal basis for building a load generator, as it is asynchronous and can be used to simulate many thousands of connections (see the Cometd Load Tester for a good example of a realistic load generator).
I wonder how to determine the How to determine the number of thread for each user in Jetty.
Since I don't know how to test and which tool should I use.
Any hint will be appreciated.
Please.
The number of threads you will require depends on the kind of load your application will need.
A typical web page is currently about 30 resources totaling about 40 MB that need to be requested for the web page to be "complete" and can be rendered fully.
If you use HTTP/2, which will get data faster, you will have a greater demand on the thread pool than if you use HTTP/1.1.
But lets say you need Jetty to supply data to a mobile chess playing app, less connections overall, you now have a much lower demand of threads per user.
The advice is as it always is, monitor your applications behavior, now, and in testing, and in QA, and in production. Learn the behavior, and adjust your configuration accordingly.
It is impossible to "do the math" and set the perfect configuration up front without monitoring. You will be adjusting these configurations over time as your application evolves and you learn more about the behaviors within your application.
I created and ran a simple SpringBoot application to accept some requests. I used jconsole to check the Heap Memory Usage and I saw this periodic increase followed by GC, I don't know the reason for the increases. Are there any Objects keep being created (because I think the instances are imported to container when the application starts)?
Spring boot has background processes which may consume your resources despite on absent requests to your app, for example jobs, etc.
Those spikes are expected and regular for any Java app based on any more less complex framework. GC algorithm depends on your jvm version, but could be overridden. The graph shows normal situation, from time to time memory consumed for some activities and after some time GC wake up and do the cleaning.
In case if you want to check what exactly objects caused memory spike you may try to use Java Flight Recorder or regular heap dump analysis using Eclipse memory analyser.
For current case Java Flight Recorder would be more convenient and suitable.
I have a spring boot webflux application which by default uses netty.
One of the business requirements that we have mandates that requests should time out within 2 seconds.
When very few requests are sent to the app, everything is fine but when the request load is increased (Like over 40 or 50 concurrent per second by Jmeter) sometimes all of them time out due to each taking longer than the 2-second threshold.
I have spent a long time reading things online and looking into what could be causing this issue but with no success. When requests are sent concurrently most end up taking a long time and the problematic part is where an external HTTTP request is made to other microservice. All my tests are local and I have tested the microservices and they seem fast enough to handle a big load so the microservices themselves are not the issue.
I know that netty uses event loop and does not create a thread per request.
I believe there are likely synchronous tasks that are blocking those few netty threads. For this reason I have done massive refactoring and have ".publishOn(Schedulers.boundedElastic())" or ".subscribeOn(Schedulers.boundedElastic())" in the Mono reactive chains. After the refactoring Most of the operations seem to be running on elastic threads and not the "reactor-http-nio-x" (According to the logs) but doing so has not helped the main issue and the problem still exists.
It will be a huge help if someone could direct me to what I should be doing. At this point, I have no more improvements to make, and think I might have been looking at this the wrong way and my approach has not been correct.
I have not attached any code sine the application is big and I do not still know where the actual problem lies.
I've encountered the same problem. I've didn't find the root cause of this, but when I switched from WebClient to RestTemplate with dedicated thread pool per client (external service) then the problem was solved. I've run a blockhound to find if I block somewhere in the stream, but it didn't find anything. I've also tried deploying my application with increased number of NIO worker thread pool (by default it's equal to cores number) and there was some improvement, but after all RestTemplate yielded the best performance. So I'm still on Webflux stack, but I don't use WebClient anymore and the performance on high load is fine.