I have deployed my application in tomcat 7. I am using the default number of threads in thread pool. When I am running the performance testing for 500 concurrent users the application is not responding. I am using vCPU-8 and 16GB RAM EC2 instance.
Can any one suggest how to decide the number of threads in the thread pool? Also Is it possible to allocate fixed number of threads for a specific url prefixes?
Please note that it is not possible to horizontal scale the infrastructure because of limitations in the framework.
Related
I have a microservice using spring boot 2.7.0 with embedded NIO tomcat. The application is responsible for receiving requests and for each request it makes 6 parallel remote calls waits at most 2 seconds for response from any of the 6 requests.
While performance testing this microservice using jmeter I observed that the CPU remains under-utilised around 14-15% but the microservice's response time increases to more than a minute. Typically it shouldn't be more than 2-3 seconds.
There are 3 thread configurations in my microservice:
Tomcat threads here I tried various configuration of maxthreads, maxconnection,accept-like (5000,30000,2000), (500,10000,2000), (200,5000,2000) but the CPU is always under-utilised. Here are the properties I am changing
server.tomcat.max-threads=200
server.tomcat.max-connections=5000
server.tomcat.accept-count=2000
server.connection-timeout=3000
For each request received we create a ForkJoinPool with parallelism as 6 to make the 6 remote calls. We tried using an ExecutorService too with different configuration like newSingleThreadExecutor,newCachedThreadPool,newWorkStealingPool. Also increased pool size to around same as maxThreads of tomcat and beyond but the result was same CPU still underutilized but microservice taking more than a minute to respond.
On logging the active thread count here we saw that no matter how much thread pool size or tomcat maxthreads we increased the, active thread count went upto 300 then start declining. We tried with a 4core 8GB system and 8core 16GB system results were exactly same
For making remote calls we use spring rest template with maxConnTotal and maxConnTotalPerRoute same as maxthreads of tomcat. maxConnTotal and maxConnTotalPerRoute are same because all 6 remote calls are to the same server.
Here are the jmeter parameters used -GTHREADS=1000 -GRAMP_UP=180 -GDURATION=300
There are 3 instances of this microservice running, roughly after 2-2.5 minutes after jmeter starts, all 3 instance's response time goes beyond a minute for all requests while CPU remains at 14-15% only. Could someone please help figure out what CPU is not spiking if CPU would spike to 35% then autoscaling would kick in but since CPU is under-utilised no scaling is happening
Use a profiler tool like VisualVM, YourKit or JProfiler to see where your application spends the most time
CPU is not the only possible bottleneck, check Tomcat's connection pool utilization as it might be the case the requests are queuing up, memory usage, network usage, database pool usage, DB slow queries log and so on. If you don't have a better monitoring software or an APM tool in place you can consider using JMeter PerfMon Plugin
We replaced RestTemplate for remote calls with WebClient and introducted WebFlux Mono to make the complete request non-blocking. The request itself now returns our response wrapped in Mono. It solved our issue now there is no idle time as threads are not blocked on IO rather they are busy serving other requests.
We have a Quarkus Rest service, and the client is using org.apache.http.impl.conn.PoolingHttpClientConnectionManager with following settings
connMgr.setMaxTotal(20);
connMgr.setDefaultMaxPerRoute(6);
How would from service we can check if the service support maximum 20 connections?
By default what is the maximum connection allowed in quarkus?
As far as I know quarkus is not limited you can configure a limit with this property but I would not suggest going over 100 connections never, because then you might run into the memory limits or cpu limits. This is why you replicate or scale your backends, so instead of having one jvm handling 150 connections, you have three smaller jvms handling 50 connections each one so you gain some high availability and fault tolerance.
If you want to test the behaviour in concurrency of your application you can always run a load test with Jmeter or other tools which will allow to simulate the load that you want and you would be able then to check the response time of your backend or if you run into resource bottlenecks or other issues.
If I have a server with 1 core, how many puma workers, threads and what database pool size is appropriate?
What's the general thumb here?
Not an easy answer.
The two main sources of information are:
Puma github repository (the authors' point of view)
Heroku's web page (the main big user's point of view)
Unfortunately they are inconsistent mostly because heroku has different deployment metrics and terminology.
So I ended up following the puma repository guidelines which says:
One worker per core
Threads to be determined in connection with RAM availability and application and
Threads = Connection Pool
So the number of threads is mostly a try and check operation.
OS: Windows Server 2012 Standard
IIS: 8.0.9200.16384
Processor: 4x Xeon 2.67Ghz CPU
RAM: 40GB
Problem:
We have recently enabled IIS's AutoStart feature, since doing so our start up time for the application pools has gone up considerably. The application pool appears to be running but it seems to ramp up its CPU usage to the maximum 25% for about 30 minutes and the websites running in that pool don't respond until this has completed. We have checked the event log and there doesn't appear to be any faults. We have checked the logging in our preload function and this appears to only take about 60-90 seconds.
How can we diagnose what is causing the delay in the application pools starting up?
Background:
We are serving up multiple copies of the same ASP.Net MVC3 application, from multiple application pools (20 sites per pool). We have approximately 8 pools service up 160 sites We have IProcessHostPreloadClient built which preloads some settings from the database when the sites starts up. We have a second server with the same basic specs but only 3 pools of 20, which only takes approx 5 minutes per pool to start up.
For anyone interested here is what we did to resolve/mitigate the issue:
Break up our sites into smaller groupings per application pools (this reduces the startup time per application pool). We went with 10 sites per pool.
Change to using the IIS 8 Application Initialize 'PreloadEnabled' option rather than the serviceAutoStartProvider for site initialization.
When deploying new code, don't restart the application pools, instead use the app_offline.htm feature to unload the application and restart it.
The app_offline.htm feature is the key one for us, this means we are able to deploy new versions of our software with out stopping and starting the application pools and incurring the start up time penalty. Also incrementally restarting application pools help reduce the strain on the CPU which meant we got a consistent start up time for each pool. This is only required when we do an IIS reset or server restart (rarely).
I'm running compojure on Heroku. They have a limit of a 100 threads per process. So when I go over that limit, I get: java.lang.OutOfMemoryError: unable to create new native thread.
Compojure is using the jetty ring adapter. Is there away of configuring the the server to only accept a hundred threads to the servlet at a time?
The solution comes from Chris Perkins over at the compojure google group.
(run-jetty app {:configurator #(.setThreadPool % (QueuedThreadPool. 5))})
This initializes a QueuedThreadPool (with a concurrent limit of five threads) to the jetty instance, before it starts.