CPU usage going 100% on making any transaction - performance

Question

Recently I came up across strange scenario where I could see the application slows down after some point of time due to which I am unable to load application.
So I started monitoring the server and found that for some of the scenario's the GC activity is going more than 60% and it is impacting directly on application perfomance so in order to fix it I tried to increase the heap memory to 2.5G from 2GB but still I could find same issue can anyone help me is increasing Heap size further is an ideal approach or something else I can do.
System Ram Capability : 6 GB
Following is VisualVm monitor screenshot:
Kindly let me know if further info is required for analysis.
Thread Dump:
http-apr-8080-exec-15" - Thread t#110
java.lang.Thread.State: RUNNABLE
at java.util.regex.Pattern$BnM.match(Pattern.java:5464)
at java.util.regex.Matcher.search(Matcher.java:1248)
at java.util.regex.Matcher.find(Matcher.java:637)
at java.util.regex.Matcher.replaceAll(Matcher.java:951)
at java.lang.String.replace(String.java:2240)
at com.srik.security.SecureTrace.maskOther(SecureTrace.java:135)
at com.srik.security.SecureTrace.maskJsonData(SecureTrace.java:102)
at com.srik.mbweb.manager.MUSR.printResponse(MUSR.java:261)
at com.srik.mbweb.manager.MUSR.invokeService(MUSR.java:212)
at com.srik.mbweb.manager.MU.invokeCacheableService(MU.java:30)
at com.srik.mbweb.manager.MU$$FastClassBySpringCGLIB$$45933c60.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.cache.interceptor.CacheInterceptor$1.invoke(CacheInterceptor.java:52)
at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:345)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:408)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:327)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at com.srik.mbweb.manager.MU$$EnhancerBySpringCGLIB$$36289a8f.invokeCacheableService(<generated>)
at com.srik.mbweb.manager.ChannelUnitedServiceManagerProxy.invokeService(ChannelUnitedServiceManagerProxy.java:35)
at com.srik.mbweb.controller.MobileUnitedController.processRequest(MobileUnitedController.java:32)
at com.srik.mbweb.controller.MobileUnitedControllerBase.invokeService(MobileUnitedControllerBase.java:58)
at sun.reflect.GeneratedMethodAccessor216.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.srik.statemachine.web.AbstractStateMachineAwareFilter.doFilterInternal(AbstractStateMachineAwareFilter.java:63)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.srik.mbweb.common.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:137)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.srik.mbweb.common.StatelessCSRFFilter.doFilterInternal(StatelessCSRFFilter.java:51)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.srik.mbweb.common.LoggerFilter.doFilterInternal(LoggerFilter.java:30)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.srik.mbweb.multitenant.filter.FITenantFilter.doFilterInternal(FITenantFilter.java:110)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at com.srik.mbweb.multitenant.filter.FITenantFilter.doFilterInternal(FITenantFilter.java:110)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1100)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:687)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2521)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2510)
- locked <1e41e77c> (a org.apache.tomcat.util.net.AprEndpoint$AprSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- locked <5896885f> (a java.util.concurrent.ThreadPoolExecutor$Worker)
SecureTrace method:
private static Pattern PWD_XML_PATTERN = Pattern.compile("^(<.*?(?:(?i)Aomn|Amn|amn|aamn)>)(.*?)(<\\/.*?(?:(?i)Aomn|Amn|amn|aamn)>)");
public static String maskData(String data) {
try {
for(Matcher m = PWD_XML_PATTERN.matcher(data); m.find(); data = m.replaceAll("AA****BB")) {
;
}
} catch (Exception var2) {
var2.printStackTrace();
}
return maskOther(data);
}
private static String maskOther(String data) {
String mask = null;
int groupLen = false;
Matcher matcher = null;
try {
matcher = MASK_PATTERN.matcher(data);
while(matcher.find()) {
mask = "****";
String group = matcher.group();
int groupLen = group.length();
if (groupLen > 5) {
mask = mask + group.substring(groupLen - 4);
data = data.replace(group, mask);
}
}
} catch (Exception var5) {
; }
return data;
}

Answer 1

It’s not clear, how much the posted code relates to the spikes in CPU usage at all, as it looks like you’ve made a snapshot when the CPU load is close to zero. You should use the sampler/profiler to gather data about the actual bottlenecks.
But it’s worth considering that spikes of 100% CPU usage are not unnatural. A single CPU core doesn’t know other modes than being either, idle (0%) or processing code (100%). Any in-between number shown by a tool is the result of smoothing multiple values of 0% and 100%. When seeing spikes of 100%, it just means that an “executing code” phase lasted longer than the time window used for smoothing.
But the garbage collector activity looks high indeed and the memory usage pattern seems to fit. So you may test how the behavior changes when giving it more heap memory. But don’t use increments as small as 25%, just to get ambivalent results. When you have 6 G available, double the heap to 4 G or even raise to 5 G. If you still see the same pattern, you have a problem. Check that there are no explicit System.gc() calls or similar in your code. If that large heap memory solved the problem, search for the best trade off between that number and the old heap size (mind how Binary Search works).
Still, the code you’ve posted bears problems. First of all, the questionable try … catch blocks. Since in this code not involving external conditions any exception is an indicator of a programming error, you should solve any existing bug and not catch exceptions.
Regarding performance, you’re doing a lot of repeated work.
In maskData, you’re executing the following loop:
for(Matcher m = PWD_XML_PATTERN.matcher(data); m.find(); data = m.replaceAll("AA****BB"))
First, you’re calling m.find(), which is fine if there’s no match, but if there is a match, you’re calling data = m.replaceAll("AA****BB") which is a self-contained operation, i.e. not using the result of the previous find(). After that, you’re calling find() again, which is pointless as the Matcher still references the original string. Since replaceAll didn’t reset the matcher, you’re not running into an infinite loop but always getting false.
So instead, use
public static String maskData(String data) {
return maskOther(PWD_XML_PATTERN.matcher(data).replaceAll("AA****BB"));
}
Even worse, the maskOther method is performing a replacement loop, but instead of using the Matcher to perform the replace operation, it is calling replace on the string for each match. This can easily run towards quadratic time complexity in respect to the number of matches, multiplied with the source string length as the replace operation creates a new string each time.
But since replace also replaces all occurrences of the search string (the difference to replaceAll is that it doesn’t do regex patterns), it depends on the actual pattern, whether subsequent iterations of the while(matcher.find()) loop will process matches that have been replaced already.
A straight-forward linear replacement operation with the same result as your method¹ may look like:
private static String maskOther(String data) {
Matcher matcher = MASK_PATTERN.matcher(data);
if(!matcher.find()) return data;
StringBuilder sb = new StringBuilder(data.length()*2);
int pos = 0;
do {
if(matcher.end() - matcher.start() <= 5) continue;
sb.append(data, pos, matcher.start()).append("****");
pos = matcher.end() - 4;
} while(matcher.find());
return pos == 0? data: sb.append(data, pos, data.length()).toString();
}
¹ as long as the results of your replace operation do not create new matches. Otherwise, the results of your method become almost unpredictable.