We are running a spring boot based service with which, we are having GC issues on running perf tests. When we looked at the heap dump on Eclipse plugin "Memory Analysis Toolkit" (MAT), we found below:
One instance of "com.fasterxml.jackson.databind.ObjectMapper" loaded by "org.apache.catalina.loader.WebappClassLoader # 0x1000050b8" occupies 2,057,443,904 (93.15%) bytes. The memory is accumulated in one instance of "com.fasterxml.jackson.databind.ser.SerializerCache" loaded by "org.apache.catalina.loader.WebappClassLoader # 0x1000050b8".
Keywords
com.fasterxml.jackson.databind.ser.SerializerCache
com.fasterxml.jackson.databind.ObjectMapper
org.apache.catalina.loader.WebappClassLoader # 0x1000050b8
Not really sure what's going on here. Why is SerializerCache holding onto so many objects in Map, and, how to clear it out.
Using
SpringIO: 2.0.6.RELEASE
The SpringIO 2.0.6.RELEASE uses jackson version 2.6.7, which, incidentally, doesn't show up on the Jackson release page of master branch (https://github.com/FasterXML/jackson-databind/blob/master/release-notes/VERSION).
I upgraded my application to use jackson version 2.8.1 and it fixed the issues.
Related
We have a Spring-boot REST application running on 3 production machines. A recent update from Spring-boot 2.1.8 to 2.2.2 has shown an initial increase of CPU by at least double. This load then increases over time whereas the old version stays steady.
I have managed to narrow this down to 2.2.x as building with 2.1.11 is ok, but 2.2.0 shows the problem.
To give an idea of scale, the old version stays at around 6% regardless of load, whereas the new version starts at around 15% and gradually increases to over 100% after about 10 hours.
I can see the initial rise with an identical build, only changing the Spring-boot version. The application uses spring-boot-starter-web and spring-boot-starter-actuator.
Any ideas? Should I raise this over at https://github.com/spring-projects/spring-boot/issues ?
This is very likely to be linked to a bug in Spring Framework that was fixed in Spring Framework 5.2.6 (or Spring Boot 2.2.7). There was a memory leak in case of concurrent requests/responses with the same media type.
See the dedicated issue as well as a report sent by a developer with lots of details. Note that this happens with both MVC and WebFlux.
We've seen this issue in some of our services, but upgrading to 2.2.7 looks to have resolved this for one (stable for two weeks).
We're starting to roll this out to more services in the hope that it can be rolled out everywhere, so might be worth trying this out?
Is there a way for me to track garbage collection of my Java application using Elastic APM and the associated Java APM agent?
I'm using Spring Boot, if that makes a difference.
Out-of-the-box I'm able to see the heap and non-heap memory utilization, but I'm not sure if there is also a way to view garbage collection.
The JVM GC metrics tracked right now are jvm.gc.alloc, jvm.gc.time, and jvm.gc.count.
If you are looking for additional ones, which ones would those be? And could you open an issue with the details.
Please import from saved objects option - https://github.com/elastic/apm-contrib/blob/master/apm-agent-java/dashboards/java_metrics_dashboard_7.x.json
I would like to ask why does the spring classloader load java classes multiple times when <context:load-time-weaver aspectj-weaving="on" /> is used in xml config?
I can see spring is using
org.springframework.context.support.ContextTypeMatchClassLoader$ContextOverridingClassLoader
classloader, which, as I read in documentation, creates new classloader instance for each loaded class. In our current project this results in 11 loaded classes of the same type - 1 using parent classloader and 10 more using ContextOverridingClassLoader (each loaded in its own). What could be the cause of this? If we startup many applications in parallel, these duplicate classes eat up too mach permgen memory (resulting in crash). We could just increase permgen memory of course, but I was curious if there is anything else to do.
As soon as I remove this configuration parameter, spring loads all classes only once. I checked this using -XX:+TraceClassLoading VM option and heapdumps.
We are using Spring 3.2.4 and AspectJ 1.7.4
Update:
After I upgraded to Spring 4.2.1, each class is now loaded 15 times. Could it be somehow connected to spring aspects?
We ended up calling GC after application context initialization, thus reducing the amount of used memory during parallel startup of many applications (a good tradeoff to longer application startup) as each application cleans up after init.
Our Glassfish 4 Application Server builds up heap space and hangs while testing high loads (200 parallel connections). We used different garbage collection settings while testing.
The application runs on a clustered Glassfish Server with one Administration Domain and two instances using --availabilityenabled=true to share session data. But we can reproduce the Heap building up on non clustered Glassfish Servers (running on our desktops), too.
We are using the following Spring/Hibernate Versions:
spring 4.0.1 (including webmvc, aop, context, beans, binding, jdbc, web, tx)
spring-security 3.2.0
hibernate 4.3.1 (core, entitymanager, validator)
jsp 2.2
elasticsearch 1.0.0
Stiuation
While normal testing the Heap starts to load up. Garbage collection is trying to do its job but the used Heap still rises.
(Whish I could post images...) you would see a rise of approx. 300M in 20 hours.
When using JMeter to simulate higher load the Heap Usage starts to rise. After 2 hours of 200 simultanious connections the Heap rises to 6G (from 0.5G) and it continues to rise till 8G is reached (approx. 4 hours in). Then bad things happen.
When performing test for 2 hours only (reaching 6G of Heap) the heap does not shrink, even when performing manual GC or leaving the Glassfish without new connections for 24 hours. It stays at 6G.
Heap dump
I took a Heapdump after 2 hours of testing:
class instances
java.util.ArrayList 19.904.237
java.lang.Object[] 15.851.496
org.jboss.weld.util.collections.ArraySet 9.192.068
org.jboss.weld.bean.interceptor.WeldInterceptorClassMetadata 9.188.347
java.util.HashMap$Entry 5.546.603
java.util.Collections$UnmodifiableRandomAccesList 5.331.810
java.util.Collections$SynchronizedRandomAccesList 5.328.892
java.lang.reflect.Constructor 2.669.192
com.google.common.collect.MapMakerInternalMap$StrongEntity 2.667.181
com.google.common.collect.MapMakerInternalMap$StrongValueReference 2.667.181
org.jboss.weld.injection.AbstractCallableInjectionPoint$1 2.664.747
org.jboss.weld.injection.ConstructorInjectionPoint 2.664.737
org.jboss.weld.injection.producer.BeanInjectionTarget 2.664.737
org.jboss.weld.injection.producer.DefaultInstanciator 2.664.731
Current Configuration
Java: Java(TM) SE Runtime Environment, 1.7.0_51-b13
JVM: Java HotSpot(TM) 64-Bit Server VM, 24.51-b03, mixed mode
Server: Server GlassFish Server Open Source Edition 4.0
Our current settings are:
-XX:+UnlockDiagnosticVMOptions
-XX:+UseParNewGC
-XX:ParallelGCThreads=4
-XX:+UseConcMarkSweepGC
-XX:MaxPermSize=1024m
-XX:NewRatio=4
-Xms4096m
-Xss256k
-Xmx8G
-javaagent:/opt/glassfish4/glassfish/lib/monitor/flashlight-agent.jar
[...]
Server Hardware
One server has 14 (including hyperthreading) CPU Cores, 32GB of ram and runs Glassfish and ElasticSearch. The Load never reaches 5 till Heap is nearly full and excessive garbage collecting kicks in. Load then rises massively.
What are we doing wrong here?
Weld, which is the implementation of CDI in Glassfish has a known memory leak in the version that ships with Glassfish 4. See: https://community.jboss.org/message/848709. You can replace the weld jar file to upgrade to version 2.0.5.Final. This will sorta fix the situation in that you will no longer see "org.jboss.weld.bean.interceptor.WeldInterceptorClassMetadata" classes grow without bound in your heap dumps.
There is another memory leak due to using tag files with parameters with CDI enabled (which is generally the case since GlassFish 4 will eagerly enable CDI by default even if you don't use it or have a beans.xml). As a workaround disable CDI or use a jsp segment include instead of a tag. See https://www.java.net/forum/topic/glassfish/glassfish/glassfish-4-memory-leak
I have a web app running on Tomcat 6.0.35, which makes use of Spring 3.1.2, Hibernate 4.1.8 and MySQL Connector 5.1.21.
I have been trying to figure out what is causing Tomcat to keep running out of memory (Perm Gen) after a few redeploys.
Note: Don't tell me to increase Tomcat's JVM memory because that will simply postpone, the problem
Specifically, I made use of the VisualVM tool, and was able to eliminate some problems, including some mysql and google threads issues. I was also able to discover and fix a problem caused by using Velocity as a singleton in the web app, and also not closing at the correct time/place some thread local variables I was having. But I still am not completely able to eliminate/figure out this Hibernate issue.
Here is what I'm doing:
Deploy my webapp from my development IDE
Open a tomcat manager window in my browser
Start VisualVM and get the HeapDump on the tomcat instance
Go the tomcat manager and redeploy my webapp
Take another HeapDump in VisualVM
My first observation is that the WebappClassLoader for the original webapp is not garbage collected.
When I scrutinize the retained objects from the second HeapDump, the class org.hibernate.internal.SessionFactoryImpl features prominently which leads me to believe that it IS NOT being destroyed/closed by Spring or something along those lines (and hence the WebappClassLoader still having a reference to it).
Has anyone encountered this problem and identified the correct fix for it?
I don't currently have an idea what could be amiss in your setup but what I know is that using Plumbr you'll most likely find the actual leak(s).