I would like to ask why does the spring classloader load java classes multiple times when <context:load-time-weaver aspectj-weaving="on" /> is used in xml config?
I can see spring is using
org.springframework.context.support.ContextTypeMatchClassLoader$ContextOverridingClassLoader
classloader, which, as I read in documentation, creates new classloader instance for each loaded class. In our current project this results in 11 loaded classes of the same type - 1 using parent classloader and 10 more using ContextOverridingClassLoader (each loaded in its own). What could be the cause of this? If we startup many applications in parallel, these duplicate classes eat up too mach permgen memory (resulting in crash). We could just increase permgen memory of course, but I was curious if there is anything else to do.
As soon as I remove this configuration parameter, spring loads all classes only once. I checked this using -XX:+TraceClassLoading VM option and heapdumps.
We are using Spring 3.2.4 and AspectJ 1.7.4
Update:
After I upgraded to Spring 4.2.1, each class is now loaded 15 times. Could it be somehow connected to spring aspects?
We ended up calling GC after application context initialization, thus reducing the amount of used memory during parallel startup of many applications (a good tradeoff to longer application startup) as each application cleans up after init.
Related
We have 40+ spring boot apps and when we try to start all of them together parallel, it takes about 9 to 10 minutes. And we notice that CPU usage is always 100% throughout this entire duration.
After all apps come up successfully and registered with Eureka, CPU usage is back to normal (on average ~30-40% CPU usage after startup).
It seems each spring boot app is taking at least about 15-20 seconds to startup, which we are not happy with since application is relatively small to start with.
We also disabled spring boot auto-configuration so to make sure only required "matching" classes are loaded at start up by spring boot. And we only gained about 1 or 2 seconds at startup after this change.
We seem to have enough system resources with 8 core CPUs and 32 gb of memory on this VM.
Spring boot version is 1.3.6.RELEASE.
Is it something to do with Spring boot? Because even when we startup single spring boot app it spikes CPU to 70-80% usage. Your help is very much appreciated!
This is more of how many beans and Auto Configurations that get executed while the application being started.
For even a simple web application along with JPA, there is a webcontainer and its thread pools, DataSources initializations and many more supporting beans and auto configurations that need to get initialized. These are some serious resource taking actions and they all are rushed at the start of the application to get application booted as soon as possible.
Given that you are starting 40+ apps like these simultaneously, the server will have to pay its toll.
There are ways you can improve the application boot time.
Remove unnecessary modules and bean definitions from your application. Most common mistake a developer makes is to include a spring-boot-starter-web when the application doesn't even need a web environment. Same goes for other starter modules.
Make use of Conditional Bean definitions with the use of #ConditionalOnMissingBean #ConditionalOnProperty #ConditionalOnClass #ConditionalOnBean #ConditionalOnMissingClass #ConditionalOnExpression. This might backfire if you make spring to check for beans with lots of conditions.
Make use of spring profiles. If you don't want a specific set of beans not to be part of that running instance you can group them into a profile and enable them or disable them
Configure initial number of threads a web container can have. Same goes for Datasources. Initiate your pool with only required number of active threads.
Using lazy-initialization for beans by annotating your classes or beans with #Lazy. This annotation can be per bean or against an entire #Configuration.
If that doesn't satisfy your needs, you can always throttle the CPU usage per process with commands like nice or cputools.
Here is an article around cputools.
The project I'm working on is considerably large. While trying to get load time weaving working for this spring project, I was instructed to use both the spring-instrument javaagent as well as the aspectjweaver javaagent. However, I notice that when using the aspectjweaver agent, my launch time shoots up 4-6 fold. I can also see identical weave messages 4-6 times coming from ContextOverridingClassLoader.
If I remove aspectjweaver however, and only use spring-instrument, I notice my launch time decrease dramatically with only a single weave message per join point coming from AppClassLoader.
The only issue being that some specific classes are not woven (I found that this is due to the spring application context not yet being loaded before the faulty classes are loaded by the class loader, as spring is the mechanism that enables the weaving). I've found a solution of my own by creating a custom javaagent which enables weaving in the same manner that spring-instrument does, only it does so in the premain rather than on application context load. It now weaves all the classes and in reasonable time.
However, I'd prefer not to go down this hacky road as I can only presume that the two agents were designed the way it is for a reason.
I wonder if anyone else has seen a similar issue with the aspectjweaver javaagent and if someone might know why that agent is so slow compared to just using spring-instrument.
If the answer interests anyone, I've figured out the issue.
Spring uses a temporary classloader ContextOverridingClassLoader to get metadata about the bean classes prior to actually loading them into the context.
The spring-instrument javaagent (or more accurately, the spring framework code which may or may not use the spring-instrument javaagent) specifically only weaves classes loaded by the classloader used to load the application context.
Code inside of InstrumentationLoadTimeWeaver$FilteringClassFileTransformer:
if (!this.targetClassLoader.equals(loader)) {
return null;
}
return this.targetTransformer.transform(
loader, className, classBeingRedefined, protectionDomain, classfileBuffer);
On the other hand, aspectjweaver does not have such a filtering mechanism and so will weave even those classes loaded by spring's temporary ContextOverridingClassLoader. Fortunately, aspectjweaver has an essentially undocumented system property (or at least I was unable to find any documentation on this) called aj.weaving.loadersToSkip. By setting this to:
-Daj.weaving.loadersToSkip=org.springframework.context.support.ContextTypeMatchClassLoader$ContextOverridingClassLoader
I was able to skip weaving for that classloader and speed up the loading of my application context tremendously.
Incidentally, I've found that both the spring-instrument and aspectjweaver ultimately both use ClassPreProcessorAgentAdapter to weave the classes, and thus it is probably not necessary to use both agents (aspectjweaver will weave a superset of the classes that spring-instrument will). However, depending on your configuration, the application might complain about the missing agent at startup so you might as well include it (at the cost of some additional unnecessary overhead).
We are running a spring boot based service with which, we are having GC issues on running perf tests. When we looked at the heap dump on Eclipse plugin "Memory Analysis Toolkit" (MAT), we found below:
One instance of "com.fasterxml.jackson.databind.ObjectMapper" loaded by "org.apache.catalina.loader.WebappClassLoader # 0x1000050b8" occupies 2,057,443,904 (93.15%) bytes. The memory is accumulated in one instance of "com.fasterxml.jackson.databind.ser.SerializerCache" loaded by "org.apache.catalina.loader.WebappClassLoader # 0x1000050b8".
Keywords
com.fasterxml.jackson.databind.ser.SerializerCache
com.fasterxml.jackson.databind.ObjectMapper
org.apache.catalina.loader.WebappClassLoader # 0x1000050b8
Not really sure what's going on here. Why is SerializerCache holding onto so many objects in Map, and, how to clear it out.
Using
SpringIO: 2.0.6.RELEASE
The SpringIO 2.0.6.RELEASE uses jackson version 2.6.7, which, incidentally, doesn't show up on the Jackson release page of master branch (https://github.com/FasterXML/jackson-databind/blob/master/release-notes/VERSION).
I upgraded my application to use jackson version 2.8.1 and it fixed the issues.
I have a Spring-WS web service that has three issues:
Slow startup time
Slow generation of the dynamic WSDL
Heavy usage of PermGen (app has to be 1.6 compatible)
Currently, the spring-ws-servlet.xml file has several <context:component-scan> elements for autowired dependencies. Two of these scan nearly everything in two external libraries containing Hibernate DAO and Entity classes. Similarly, the Hibernate session factory bean scans a large number of entities from these two libraries.
So, my questions:
Obviously, we would see at least some performance improvement by limiting the scope of the <context:component-scan> elements. But really, would it be that much?
Similarly, would I see improvements by limiting the scope of what Entities are scanned by the session factory?
Making these changes will NOT be a quick process (alter code, test, etc). Therefore, if anyone can add their wisdom, I would greatly appreciate it.
Actually I am developing a spring ws application on Google Cloud and I also have the same problem with slow start up time. The biggest difference that I have notice was when I have moved to aspectj compile time weaving using aspectj-maven-plugin. If you haven't done this yet try this one. The result may be vary depends on your code and deployment environment. On the cloud every file operation is much slower so this may be a reason why this work for me so well.
I've 2 applications each one uses different spring application context configuration on the same JVM, and every time i tries to run both of them together i found a problem that the last one configuration always overrides the previous one so the spring context loaded with the last one configuration, any advices how to overcome this, by letting every application runs with it's configuration without affected by other spring context.