I have a web app running on Tomcat 6.0.35, which makes use of Spring 3.1.2, Hibernate 4.1.8 and MySQL Connector 5.1.21.
I have been trying to figure out what is causing Tomcat to keep running out of memory (Perm Gen) after a few redeploys.
Note: Don't tell me to increase Tomcat's JVM memory because that will simply postpone, the problem
Specifically, I made use of the VisualVM tool, and was able to eliminate some problems, including some mysql and google threads issues. I was also able to discover and fix a problem caused by using Velocity as a singleton in the web app, and also not closing at the correct time/place some thread local variables I was having. But I still am not completely able to eliminate/figure out this Hibernate issue.
Here is what I'm doing:
Deploy my webapp from my development IDE
Open a tomcat manager window in my browser
Start VisualVM and get the HeapDump on the tomcat instance
Go the tomcat manager and redeploy my webapp
Take another HeapDump in VisualVM
My first observation is that the WebappClassLoader for the original webapp is not garbage collected.
When I scrutinize the retained objects from the second HeapDump, the class org.hibernate.internal.SessionFactoryImpl features prominently which leads me to believe that it IS NOT being destroyed/closed by Spring or something along those lines (and hence the WebappClassLoader still having a reference to it).
Has anyone encountered this problem and identified the correct fix for it?
I don't currently have an idea what could be amiss in your setup but what I know is that using Plumbr you'll most likely find the actual leak(s).
Related
Completely reworked due to new information:
Initial problem: JBoss killed itself with "OutOfMemoryError: Metaspace", when redeploying a simple Spring Boot app (2.2.6) multiple times
Using a heap dump I found that the H2 driver (1.4.200) kept a reference and leading to a problem
Afterwards I checked why tomcat does not behave like this, finding
The web application [killerApp] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
So it just works in a tomcat, because it is smart enough to notice the memory leak and preventing it. The JBoss does not do it and runs into trouble...
It looks like a standard spring boot app with H2 is misbehaving regarding the handling of unregistering the H2 driver...at least this is my take away.
I additionally found this one: GitHub - Spring Boot - Discussion about unregister managed JDBC drivers
Now I am unsure, who is at "fault"?
Me because I would need to take care of unregistering the H2 driver properly
Me because I misconfigured the Spring Boot App
The H2 driver for holding an internal state?
Spring boot for relaying on Tomcat to tidy up
Best and many thanks
Another weekend with MAT & OOMs, I now fixed the memory leaks.
First I now manually deregister the JDBC driver initiated from Spring Boot in the ContextListener (from 2.3.0 Spring boot will deregister automatically -> https://github.com/spring-projects/spring-boot/issues/21221).
This solves the first memory leak.
The other comes from io.github.classgraph. The destroy thing is added as an application shutdown hook, which is only executed when the JVM is shut down. -> not a good idea for a ever running application server. They fixed it, but springdoc-openapi-ui / org.webjars:webjars-locator-core are still using the old version with the bug. So I manually increased the classgraph version.
This solves the second memory leak...damn I never wanted to know so much about this stuff (otherwise I would program in C :D)
Problem solved :)
I have a web application that used to run fine on many web servers (tomcat, jboss, weblogic and websphere). Now, however, it has an error when deploying on WebSphere 9.
The app contains the jar javax.transaction-api-1.2. Some of its classes, e.g., javax.transaction.xa.XAResource, are also included in Java SE, but not all of them. Some are specific to Java EE and are required by some 3rd-party libraries in my app. The app is always deploying with child-first (parent-last) classloader.
WebSphere 9 throws this error during startup when the app tries to load the Oracle JDBC driver:
java.lang.LinkageError: loading constraint violation: loader "com/ibm/ws/classloader/CompoundClassLoader#7157be44" previously initiated loading for a different type with name
"javax/transaction/xa/XAResource" defined by loader "com/ibm/oti/vm/BootstrapClassLoader#422c7b1b"
Note that we aren't actually using XA transactions in the app, we are using regular transactions.
On other servers, and previous versions of WebSphere, it was never a problem. The server didn't care that we load XAResource from inside the war, even if it was previously loaded somewhere in the server. Now WebSphere 9 is different, it says that the app classloader already loaded this class from the server, but I don't know why or when did this happen.
Any idea how to solve this?
Remove the transaction API from your application. JTA 1.2 is already included in the server and provides no value in your applications. It's always risky to bring Java EE/SE APIs in a parent-last class loader unless you are 100% certain that they are technically necessary, because they can lead to issues like this one.
I can't say how this worked in previous server versions (there have been some Java-level changes in enforcing linkage issues like this), but the solution is reasonably straightforward.
At the end we did two things to solve this problem.
1) We upgraded the jta jar to version 1.3 (link here). This jar solves the problem by avoiding duplicate classes - it contains only J2EE classes and omits the J2SE classes that are already included in the JVM.
2) We upgraded WebSphere server from 9.0.0.7 to 9.0.0.11.
At the time, I suspected just upgrading the jar should suffice, but our QA had some issues with it and they also upgraded the server. Due to lack of time, we didn't investigate it further and just decided to do both.
I have a grails 3.1.7 project that uses spring security core 3.1.1 and deployed to a tomcat instance (not sure of the tomcat version).
This line appears in the log from time to time:
13-Apr-2018 13:31:20.710 SEVERE [localhost-startStop-2]
org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
The web application [my-app] created a ThreadLocal with key of
type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal#6678f8db]) and a value of type
[org.springframework.security.web.firewall.FirewalledResponse] (value
[org.springframework.security.web.firewall.FirewalledResponse#159982ef])
but failed to remove it when the web application was stopped. Threads
are going to be renewed over time to try and avoid a probable memory
leak.
I know this is part of tomcats normal operation when it checks for memory leaks and it seems that most people just choose to ignore it. That is what we have done in the past. This time we were told to 'fix it', but given that the threadlocal variable that leaked is coming from a spring security class, I am not sure what to fix or how to fix it.
So before I embark on a long detour trying to debug this, does anyone know what is going on here? Has anyone else seen this? Is it benign or do I need to do some more digging? What should I tell security to convince them to ignore it?
Any help would be much appreciated!
Turns out the team that reported the bug did indeed kill the webapp with 'kill -9'. Makes sense that tomcat cleaned up after it.
Latest project I used Spring boot, and prepare to deploy to production environment, I want to know which way to run application have better performance or have the same performance?
generate a war package and put it in a stand-alone tomcat
generate a jar package and use embedded tomcat
In addition, when publish to production environment if should to remove devtools dependency.
This is a broad question. The answer is it depends on your requirements.
Personally, I prefer standalone applications with Spring Boot today. One app, one JVM. It gives you more flexibility and reliability in regard to deployments and runtime behaviour. Spring Boot 1.3.0.RELEASE comes with init scripts which allows you to run your Spring Boot application as a daemon on a Linux server. For instance, you can integrate rpm-maven-plugin into your build pipeline in order to package and publish your application as a RPM for deployment or you can dockerize your application easily.
With a classic deployment into a servlet container like Tomcat you will be facing various memory leaks after redeployment for example with logging frameworks, badly managed thread local objects, JDBC drivers and a lot more.
Either you spend time to fix all of those memory leaks inside your application and frameworks you use or just restart servlet container after a deployment. Running your application as a standalone version, you don't care about those memory leaks because you are forced to restart in order to bring you new version up.
In the past, several webapps ran inside one servlet container. This could lead to performance degradation for all webapps because every webapp has its own memory, cpu and GC characteristics which may interfere with each other. Further more, resources like thread pools were shared among all webapps.
In fact, a standalone application is not save from performance degradation due to high load on the server but it does not interfere with others in respect to memory utilization or GC. Keep in mind that performance or GC tuning is much more simpler if you can focus on the characteristics of just one application. It gets complicated as soon as you'll need to find common denominator for several webapps in one servlet container.
In the end, your decision may depend on your work environment. If you are building an application in a corporation where software is running and maintained by operations, it is more likely that you are forced to build a war. If you have the freedom to choose your deployment target, then I recommend a standalone application.
In order to remove devtools from a production build
you can use set the excludeDevtools build property to completely
remove the JAR. The property is supported with both the Maven and
Gradle plugins.
See Spring Boot documentation.
I've been looking around for a way to run a titan server in tomcat, but I can't find any information about this.
Anyone that knows how this can be done?
Since you are asking about running "Titan Server" in Tomcat, that really just means how to run Rexster inside of Tomcat. We dropped official support for Tomcat many, many versions ago, but I believe there are still those have it deployed that way which means it is in fact possible. I guess this would also only apply to hosting the Jersey-based REST endpoints and not RexPro.
To get started I would simplify the stack and just get Rexster running in Tomcat. I would search around the gremlin-users mailing list for what people have posted on the topic, but I think that this one is the most relevant:
https://groups.google.com/forum/#!msg/gremlin-users/s0g9Sd_xjSw/LQ3_ugL680cJ
If I remember correctly the key to making things work lies in this Rexster class: RexsterApplicationProvider. Note the class comments with the sample web.xml fragment.
I suspect you just to fire up an instance of Titan with Cassandra etc when Tomcat starts?
If this is the case you can make a InitialListener in your web.xml that starts a Singleton or service that starts up the titan graph connection and then you can use it in your other servlets/whatever code base you have running.