Spring Boot Thread Pool metrics - spring

Question

we have a Spring Boot application which, as the load increases, becomes unresponsive. We have checked the CPU, RAM and Database connections and we cannot see any peak for them.
However, there are about 200 threads with the same Stack Trace:
-Spring #Service invoked here
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
I think this could be a bottleneck in some thread pools. Which are the metrics that I should check? tomcat thread pool size ? anything more specific for mvc?
Thanks a lot

Answer 1

A good metric to begin with is probably the current thread count inside the JVM.
You can use:
//only own Threadgroup
Thread.activeCount();
// all Threads, using JMX API
ManagementFactory.getThreadMXBean().getThreadCount();
Or you can use an external tool like Java VisualVM
I suggest using thread dumps to home in on the issue. You could create thread dumps every 10 seconds or so and store them until the error occurs.
If you have storage issues with this, remove older dumps, as we are only interested in the dumps surrounding the incident.
Subsequently, you can analyze the dumps using JProfiler
I also found this article regarding thread dump analyzing and this article regarding thread counting very helpful.