Completely reworked due to new information:
Initial problem: JBoss killed itself with "OutOfMemoryError: Metaspace", when redeploying a simple Spring Boot app (2.2.6) multiple times
Using a heap dump I found that the H2 driver (1.4.200) kept a reference and leading to a problem
Afterwards I checked why tomcat does not behave like this, finding
The web application [killerApp] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
So it just works in a tomcat, because it is smart enough to notice the memory leak and preventing it. The JBoss does not do it and runs into trouble...
It looks like a standard spring boot app with H2 is misbehaving regarding the handling of unregistering the H2 driver...at least this is my take away.
I additionally found this one: GitHub - Spring Boot - Discussion about unregister managed JDBC drivers
Now I am unsure, who is at "fault"?
Me because I would need to take care of unregistering the H2 driver properly
Me because I misconfigured the Spring Boot App
The H2 driver for holding an internal state?
Spring boot for relaying on Tomcat to tidy up
Best and many thanks
Another weekend with MAT & OOMs, I now fixed the memory leaks.
First I now manually deregister the JDBC driver initiated from Spring Boot in the ContextListener (from 2.3.0 Spring boot will deregister automatically -> https://github.com/spring-projects/spring-boot/issues/21221).
This solves the first memory leak.
The other comes from io.github.classgraph. The destroy thing is added as an application shutdown hook, which is only executed when the JVM is shut down. -> not a good idea for a ever running application server. They fixed it, but springdoc-openapi-ui / org.webjars:webjars-locator-core are still using the old version with the bug. So I manually increased the classgraph version.
This solves the second memory leak...damn I never wanted to know so much about this stuff (otherwise I would program in C :D)
Problem solved :)
Related
I have a web application that used to run fine on many web servers (tomcat, jboss, weblogic and websphere). Now, however, it has an error when deploying on WebSphere 9.
The app contains the jar javax.transaction-api-1.2. Some of its classes, e.g., javax.transaction.xa.XAResource, are also included in Java SE, but not all of them. Some are specific to Java EE and are required by some 3rd-party libraries in my app. The app is always deploying with child-first (parent-last) classloader.
WebSphere 9 throws this error during startup when the app tries to load the Oracle JDBC driver:
java.lang.LinkageError: loading constraint violation: loader "com/ibm/ws/classloader/CompoundClassLoader#7157be44" previously initiated loading for a different type with name
"javax/transaction/xa/XAResource" defined by loader "com/ibm/oti/vm/BootstrapClassLoader#422c7b1b"
Note that we aren't actually using XA transactions in the app, we are using regular transactions.
On other servers, and previous versions of WebSphere, it was never a problem. The server didn't care that we load XAResource from inside the war, even if it was previously loaded somewhere in the server. Now WebSphere 9 is different, it says that the app classloader already loaded this class from the server, but I don't know why or when did this happen.
Any idea how to solve this?
Remove the transaction API from your application. JTA 1.2 is already included in the server and provides no value in your applications. It's always risky to bring Java EE/SE APIs in a parent-last class loader unless you are 100% certain that they are technically necessary, because they can lead to issues like this one.
I can't say how this worked in previous server versions (there have been some Java-level changes in enforcing linkage issues like this), but the solution is reasonably straightforward.
At the end we did two things to solve this problem.
1) We upgraded the jta jar to version 1.3 (link here). This jar solves the problem by avoiding duplicate classes - it contains only J2EE classes and omits the J2SE classes that are already included in the JVM.
2) We upgraded WebSphere server from 9.0.0.7 to 9.0.0.11.
At the time, I suspected just upgrading the jar should suffice, but our QA had some issues with it and they also upgraded the server. Due to lack of time, we didn't investigate it further and just decided to do both.
When I run my Spring Boot application (from within Intellij IDEA) and either trigger a stop, restart or automatic redeploy of the application; something is issuing a shutdown command to my HSQLDB.
I run a HSQLDB (v.2.3.4) in Server mode from an external terminal window.
HSQLDB Server log at the moment I restart or stop my Spring Boot application:
[Server#4f023edb]: Initiating shutdown sequence...
[Server#4f023edb]: Shutdown sequence completed in 101 ms.
[Server#4f023edb]: 2017-05-05 21:47:01.878 SHUTDOWN : System.exit() is called next
This is of course very annoying since I have to go through the hassle of manually bringing up my HSQLDB every time I redeploy the application. How do I prevent this from happening, or an explanation of what is actually going on.
This only seems to happen when running the Spring Boot application from within Intellij IDEA, if I start the Spring Boot application-jar from a terminal window and issue shutdown Ctrl+C, then HSQLDB is not affected.
Turns out the reason why I only experience this problem when running from within Intellij IDEA is because spring-boot-devtools (a maven dependency included in my project) is not packaged in the application-jar that I run from a terminal window.
Developer tools are automatically disabled when running a fully packaged application. If your application is launched using java -jar or if it’s started using a special classloader, then it is considered a “production application”. Flagging the dependency as optional is a best practice that prevents devtools from being transitively applied to other modules using your project. Gradle does not support optional dependencies out-of-the-box so you may want to have a look to the propdeps-plugin in the meantime.
spring-boot-devtools is active when running the application from within Intellij IDEA and provides a shutdown hook that will try to gracefully shutdown the database resource (amongst other things).
The shutdown hook can be disabled the following way;
SpringApplication app = new SpringApplication(MyApplication.class);
app.setRegisterShutdownHook(false); //This disables the shutdown hook
app.run(args);
This solution resolved the problem I had.
I have a web app running on Tomcat 6.0.35, which makes use of Spring 3.1.2, Hibernate 4.1.8 and MySQL Connector 5.1.21.
I have been trying to figure out what is causing Tomcat to keep running out of memory (Perm Gen) after a few redeploys.
Note: Don't tell me to increase Tomcat's JVM memory because that will simply postpone, the problem
Specifically, I made use of the VisualVM tool, and was able to eliminate some problems, including some mysql and google threads issues. I was also able to discover and fix a problem caused by using Velocity as a singleton in the web app, and also not closing at the correct time/place some thread local variables I was having. But I still am not completely able to eliminate/figure out this Hibernate issue.
Here is what I'm doing:
Deploy my webapp from my development IDE
Open a tomcat manager window in my browser
Start VisualVM and get the HeapDump on the tomcat instance
Go the tomcat manager and redeploy my webapp
Take another HeapDump in VisualVM
My first observation is that the WebappClassLoader for the original webapp is not garbage collected.
When I scrutinize the retained objects from the second HeapDump, the class org.hibernate.internal.SessionFactoryImpl features prominently which leads me to believe that it IS NOT being destroyed/closed by Spring or something along those lines (and hence the WebappClassLoader still having a reference to it).
Has anyone encountered this problem and identified the correct fix for it?
I don't currently have an idea what could be amiss in your setup but what I know is that using Plumbr you'll most likely find the actual leak(s).
the glassfish application server provides a nice monitoring REST interface.
To use it u can enable several monitorable items in the admin console, for example the EJB container. The documentation says, you can retreive EJB-statistics for every deployed application.
If you request a URL like localhost:4848/monitoring/domain1/server/applications/APPNAME/EJBNAME you will get statistics for a given EJB of the application.
Further, there is a possibility to look more deeply into each bean-method of the ejb, for example the executiontime, about which the documentation says:
"Time, in milliseconds, spent executing the method for the last successful/unsuccessful attempt to run the operation. This is collected for stateless and stateful session beans and entity beans if monitoring is enabled on the EJB container."
The problem now is, monitoring is enabled on the EJB-container (Level set to HIGH), but nothing is sampled in any bean-method in any EJB in any deployed application.
Is there something special to do in the bean and/or the glassfish ?
Thanks in advance for help,
Chris
EDIT:
Ok, I noticed something more about that behaviour:
In the server log you get a log message for each deployed EJB like that:
INFO: EJB5181:Portable JNDI names for EJB DataFetcher // ...
If I set the ejb-container monitoring level to HIGH (which is what I want to do), I get the following warning for each deployed EJB, regardless which app I deploy:
WARNING: MNTG0201:Flashlight listener registration failed for listener class : com.sun.ejb.monitoring.stats.StatelessSessionBeanStatsProvider , will retry later
I googled the warning but none of the resulst really help me enabling EJB monitoring...
This seems to be a Bug in Glassfish.
EJB Monitoring is currently not working in 3.1.2.
JIRA issue is already raised: http://java.net/jira/browse/GLASSFISH-19677
There is nothing "special" to do.
http://docs.oracle.com/cd/E18930_01/html/821-2431/abeea.html
For me it seems as if you probably enabled the monitoring option on the wrong configuration. Please double check.
To get rid of this message you can disable the monitoring on ejb container option below in the image
From Monitor Data--->Configure monitoring--->make ejb container log off
we have a WAR packaged web application which works on a development machine. It consists of Java EE 5 + Seam 2.2.1 + Hibernate 3.3.1. It uses an Oracle 10g XE database and runs on the latest Tomcat 6 using JDK 1.6.0_24. The system is a 64bit Windows 7 Pro and uses Firefox 6.
Now, we tried to migrate the application to Oracle WebLogic 10.3.5. We followed the Seam manual for specifying the datasource, deploying the JSF lib and configuring the necessary xml-Files. In parallel to that, we verified that a newly seam-gen generated application using the same datasource and having the xml files configured as stated works fine.
But if we try our own application on WebLogic, allthough the deployment works fine, when trying to open the web application in the browser, it shows that endless redirection occured. We don't see any stacktrace or log error, even when configuring log4j in WebLogic. When analyzing the request with Firebug, we see the get request to "login.seam" and then 20 times a get request for "error.seam" (or "debug.seam when setting debug mode) with the response "302 Moved Temporarily".
I worked on that all week now, trying to compare all configurations and couldn't solve it - so I hope to get any ideas what could cause that problem and how to solve it!
I could pin down the problem to starting a transaction on the JTA persistence context.
I still don't understand why I didn't see any log message or stacktrace and why this also happens on the debug/error page (that's why the endless redirect occured).
I ended up using RESOURCE_LOCAL so far, but I'm not sure if this is the right thing to do.
The alternative is disabling the seam transaction setting, but then it didn't work together with the development build which still runs on Tomcat.