I am using jmeter to load test my APi server(running on tomcat) which inturn calls a micrroservicr using thrift.(20k requests/min)
I am using new relic for monitoring . I have observed that a an abnormally high time is spent when API calls the microservice(ranging from 10-15seconds).So I observed the microservice over the same duration. The response time was almost negligible.(10-12 milliseconds)
So, I suspected probably API is queueing up the responses because it is unable to accept the rate at which its receiving response from the microservice.To address the same I doubled Xmx and Xms value of my API java application.
Still am observing the same , what could be the bottleneck which I am missing out.
Make sure that your API running on Tomcat has enough headroom in terms of CPU, Ram, Network, Disk, etc. as it might be slowing the things down. You can use JMeter PerfMon Plugin for this
Make sure that Tomcat itself is configured for high loads as the threads might be queuing up on Tomcat HTTP Connector, i.e. if threads in executor are less than the number of connections you establish - the requests will be queuing up even before reaching your API
Re-run your test using profiler tools telemetry, i.e. set up JProfiler or YourKit monitoring - this way you will learn where your API spends the most of time and what is the underlying reason
Related
I have a microservice using spring boot 2.7.0 with embedded NIO tomcat. The application is responsible for receiving requests and for each request it makes 6 parallel remote calls waits at most 2 seconds for response from any of the 6 requests.
While performance testing this microservice using jmeter I observed that the CPU remains under-utilised around 14-15% but the microservice's response time increases to more than a minute. Typically it shouldn't be more than 2-3 seconds.
There are 3 thread configurations in my microservice:
Tomcat threads here I tried various configuration of maxthreads, maxconnection,accept-like (5000,30000,2000), (500,10000,2000), (200,5000,2000) but the CPU is always under-utilised. Here are the properties I am changing
server.tomcat.max-threads=200
server.tomcat.max-connections=5000
server.tomcat.accept-count=2000
server.connection-timeout=3000
For each request received we create a ForkJoinPool with parallelism as 6 to make the 6 remote calls. We tried using an ExecutorService too with different configuration like newSingleThreadExecutor,newCachedThreadPool,newWorkStealingPool. Also increased pool size to around same as maxThreads of tomcat and beyond but the result was same CPU still underutilized but microservice taking more than a minute to respond.
On logging the active thread count here we saw that no matter how much thread pool size or tomcat maxthreads we increased the, active thread count went upto 300 then start declining. We tried with a 4core 8GB system and 8core 16GB system results were exactly same
For making remote calls we use spring rest template with maxConnTotal and maxConnTotalPerRoute same as maxthreads of tomcat. maxConnTotal and maxConnTotalPerRoute are same because all 6 remote calls are to the same server.
Here are the jmeter parameters used -GTHREADS=1000 -GRAMP_UP=180 -GDURATION=300
There are 3 instances of this microservice running, roughly after 2-2.5 minutes after jmeter starts, all 3 instance's response time goes beyond a minute for all requests while CPU remains at 14-15% only. Could someone please help figure out what CPU is not spiking if CPU would spike to 35% then autoscaling would kick in but since CPU is under-utilised no scaling is happening
Use a profiler tool like VisualVM, YourKit or JProfiler to see where your application spends the most time
CPU is not the only possible bottleneck, check Tomcat's connection pool utilization as it might be the case the requests are queuing up, memory usage, network usage, database pool usage, DB slow queries log and so on. If you don't have a better monitoring software or an APM tool in place you can consider using JMeter PerfMon Plugin
We replaced RestTemplate for remote calls with WebClient and introducted WebFlux Mono to make the complete request non-blocking. The request itself now returns our response wrapped in Mono. It solved our issue now there is no idle time as threads are not blocked on IO rather they are busy serving other requests.
We have a Quarkus Rest service, and the client is using org.apache.http.impl.conn.PoolingHttpClientConnectionManager with following settings
connMgr.setMaxTotal(20);
connMgr.setDefaultMaxPerRoute(6);
How would from service we can check if the service support maximum 20 connections?
By default what is the maximum connection allowed in quarkus?
As far as I know quarkus is not limited you can configure a limit with this property but I would not suggest going over 100 connections never, because then you might run into the memory limits or cpu limits. This is why you replicate or scale your backends, so instead of having one jvm handling 150 connections, you have three smaller jvms handling 50 connections each one so you gain some high availability and fault tolerance.
If you want to test the behaviour in concurrency of your application you can always run a load test with Jmeter or other tools which will allow to simulate the load that you want and you would be able then to check the response time of your backend or if you run into resource bottlenecks or other issues.
The web application is running on Springboot and deployed on WebLogic.
We have assigned 400 as max threads and JDBC to be 100 connections.
When we perform load testing on the web application, the performance is optimal when the load is low (the response time is less than 200ms for most of the http request that we called).
When we increase the load, we can see that the thread count increases and jdbc count also increases gradually but no where near to max. However, the response time is getting much longer and it could take more than 5 seconds to response.
CPU usage, thread count, memory, JDBC connection seems to be normal during these period.
Another observation is that during testing and we saw that the performance is degrading, we used another machine to make a http call to the server that is only retrieving text without any DB or logic, and even this simple http call will take 10s to respond. (And the server resources is still not MAX!)
So, we are wondering what keep them waiting ?
Any other possible bottleneck?
If the server doesn't lack resources like CPU/RAM/etc. only a profiler can tell you where your application spends the most time which might be in:
Waiting in a queue for next thread/db connection from the pool to be available
Slow database query
Inefficient functions/algorithms which a subject to optimization
WebLogic configuration not suitable for high loads
JVM configuration not suitable for high loads (i.e. system is doing garbage collection to often/too long)
So I would recommend re-running your test with profiler tool telemetry enabled and at the same time monitoring essential JVM metrics using i.e. JMXMon Sample Collector which can be used for monitoring your application-specific metrics as well. It's a plugin which can be installed using JMeter Plugins Manager
For a detailed approach on how ago about identifying poor thread performance I suggest you take look at the TSA Method by Brendan Gregg.
We are executing a test of Upload scenario where we are aware that the response time will be more than 5 minutes. Hence we have configured timeout in HTTP Request Defaults as well as in the Http request as 3600000 milliseconds. But still we are getting Socket Exception in Upload transaction . Could you please suggest how to handle this.
Thanks,
SocketException doesn't necessarily means "timeout", it indicates that JMeter is not able to create or access Socket connection, there are too many possible reasons, the most common are:
Network configuration of your server doesn't allow that many connections as you're trying to open, check the maximum number of open connections on your application server and operating system level.
Your application server is overloaded and cannot handle such a big load. Make sure it has enough headroom to operate in terms of CPU, RAM and especially Network metrics (these can be monitored using JMeter PerfMon Plugin)
You might be experiencing the behaviour described in JMeterSocketClosed article
Basically the same as points 1 and 2 but this time you need to check JMeter health, make sure you're following JMeter Best Practices and maybe even consider going for distributed testing
How to setup PerfMon Metrics Collector properly, I have just installed JMeter Plugins and add PerfMon on my TestPlan.
Network Latency is something JMeter measures itself, you don't need to additionally collect it
Latency. JMeter measures the latency from just before sending the request to just after the first response has been received. Thus the time includes all the processing needed to assemble the request as well as assembling the first part of the response, which in general will be longer than one byte. Protocol analysers (such as Wireshark) measure the time when bytes are actually sent/received over the interface. The JMeter time should be closer to that which is experienced by a browser or other application client.
In order to measure CPU and RAM usage
Download PerfMon Server Agent and install it on the server which you would like to monitor
Launch ServerAgent (make sure that TCP and UDP inbound/outbound traffic is allowed on port 4444 in the firewall)
Add PerfMon Metrics Collector listener to your Test Plan
Configure it like:
Run your test. Make sure it lasts longer than several seconds - you should see the CPU and Memory usage charts plotted.
Check out How to Monitor Your Server Health & Performance During a JMeter Load Test article for comprehensive instructions on PerfMon installation, configuration and usage.