We are observing Tomcat JVM heap size is reaching warning state (80%) everyday and recovering automatically sometimes and sometimes recovering post tomcat restrat.
Please guide me how to increase JVM heap size for Tomcat 7 version. I see below content in setenv.sh file of Tomcat.
CATALINA_OPTS="$CATALINA_OPTS -XX:+UseConcMarkSweepGC"
Thanks in advance.
Hello,
We are observing Tomcat JVM heap size is reaching warning state (80%) everyday and recovering automatically sometimes and sometimes recovering post tomcat restrat.
Please guide me how to increase JVM heap size for Tomcat 7 version. I see below content in setenv.sh file of Tomcat.
ATALINA_OPTS="$CATALINA_OPTS -XX:+UseConcMarkSweepGC"
Thanks in advance.
Related
I am getting java.lang.OutOfMemoryError: Metaspace exception since new deployment on production(Before this change we were using a separate jar for scheduling it was working fine but due to some network issue it was stopping again so we added scheduler and included into wildfly server with other war) env. So basically we are using wildfly 11.0.0 final server in which we have 4 war files and one of them has #scheduled - Or scheduler and it run every 10 mins. So generally we do stop the service of wildfly and start again after new war deployment, but after certain time (4to5 hours) application start slowing down and when see the console of server there i can see java.lang.OutOfMemoryError: Metaspace as below :
WARN [org.jboss.modules] (default task-11) Failed to define class com.arjuna.ats.jta.cdi.TransactionScopeCleanup$1 in Module "org.jboss.jts" from local module loader #1e802ef9 (finder: local module finder #2b6faea6 (roots: E:\Data\wildfly-11.0.0.Final\modules,E:\Data\wildfly-11.0.0.Final\modules\system\layers\base)): java.lang.OutOfMemoryError: Metaspace
ERROR [org.jboss.as.ejb3.invocation] (default task-55) WFLYEJB0034: EJB Invocation failed on component AuditLoggerHandler for method public void com.banctec.caseware.server.logger.AuditLoggerHandlerBean.publishCaseAudit(java.lang.String,com.banctec.caseware.server.helpers.SessionHolder,com.banctec.caseware.resources.Resource[],java.lang.Long) throws com.banctec.caseware.exceptions.CaseWareException: javax.ejb.EJBTransactionRolledbackException: WFLYEJB0457: Unexpected Error
So then for each operation we get similar kind of errors with java.lang.OutOfMemoryError: Metaspace
So very first i have removed plain code from #scheduler and used Executor framework where i have used 5 fixed thread pool and with this change we deployed again but again same issue is coming.
I am not sure what is causing server down again and again and getting this memory leak issue.
In all 4 war we used Spring boot 2.0.2.
Any help appreciated. Sorry for bad English.
You need to increase your heap space. And check if you have a memory leak. Please take a look at following link. http://www.mastertheboss.com/java/solving-java-lang-outofmemoryerror-metaspace-error/
You can use tools like Jprofiler to find memory leak. It works like a charm. Check out following link https://www.youtube.com/watch?v=032aTGa-1XM
I have a vaadin app where if I don't limit the RAM, the WAR will run it up to 2.5+gb in Tomcat but if I limit it to 1gb in eclipse using this the program will stay steady around 700mb(when no action)-1.2gb (when running a large dataset).
Is there a way to export this war with the memory constraint? I have other war apps on the same Tomcat server, but this one is the only one that runs rampant. Or is it better practice to create a separate virtual server and set the memory constraint in Tomcat just for it?
Eclipse runs each program in a separate JVM.
A Tomcat instance runs inside of a JVM, and so will all web apps deployed in it. You could start a second Tomcat instance (and thus a second JVM) and set the specific memory constraints for each JVM
A Grails 1.3.9 application, which is deployed with Tomcat 7 gets serious performance issues after it is used by 10 to 20 users concurrently.
CPU load increases dramatically and HTTP responses become very slow.
Second-level EhCache is used with this application.
We discovered that it helps to clear the second level cache using Melody plugin.
I tried many different setting of the EhCache (changing expiry times, using memory cache only...) but high CPU load and severe performance problems still strike after some time.
We suspect that performance issues are somehow related to second-level cache, but we were unable to find out how to resolve the problem.
We'd appreciate any suggestions to resolve this situation. Thanks.
Edit:
Memory history from Melody:
Tomcat 7 JVM arguments:
-Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties
-Djava.awt.headless=true
-Xss1G
-Xmx2G
-Xms2G
-XX:MaxPermSize=256m
-XX:PermSize=128m
-XX:+UseConcMarkSweepGC
-Dstringchararrayaccessor.disabled=true
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.endorsed.dirs=/usr/share/tomcat7/endorsed
-Dcatalina.base=/var/lib/tomcat7
-Dcatalina.home=/usr/share/tomcat7
-Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp
Our Glassfish 4 Application Server builds up heap space and hangs while testing high loads (200 parallel connections). We used different garbage collection settings while testing.
The application runs on a clustered Glassfish Server with one Administration Domain and two instances using --availabilityenabled=true to share session data. But we can reproduce the Heap building up on non clustered Glassfish Servers (running on our desktops), too.
We are using the following Spring/Hibernate Versions:
spring 4.0.1 (including webmvc, aop, context, beans, binding, jdbc, web, tx)
spring-security 3.2.0
hibernate 4.3.1 (core, entitymanager, validator)
jsp 2.2
elasticsearch 1.0.0
Stiuation
While normal testing the Heap starts to load up. Garbage collection is trying to do its job but the used Heap still rises.
(Whish I could post images...) you would see a rise of approx. 300M in 20 hours.
When using JMeter to simulate higher load the Heap Usage starts to rise. After 2 hours of 200 simultanious connections the Heap rises to 6G (from 0.5G) and it continues to rise till 8G is reached (approx. 4 hours in). Then bad things happen.
When performing test for 2 hours only (reaching 6G of Heap) the heap does not shrink, even when performing manual GC or leaving the Glassfish without new connections for 24 hours. It stays at 6G.
Heap dump
I took a Heapdump after 2 hours of testing:
class instances
java.util.ArrayList 19.904.237
java.lang.Object[] 15.851.496
org.jboss.weld.util.collections.ArraySet 9.192.068
org.jboss.weld.bean.interceptor.WeldInterceptorClassMetadata 9.188.347
java.util.HashMap$Entry 5.546.603
java.util.Collections$UnmodifiableRandomAccesList 5.331.810
java.util.Collections$SynchronizedRandomAccesList 5.328.892
java.lang.reflect.Constructor 2.669.192
com.google.common.collect.MapMakerInternalMap$StrongEntity 2.667.181
com.google.common.collect.MapMakerInternalMap$StrongValueReference 2.667.181
org.jboss.weld.injection.AbstractCallableInjectionPoint$1 2.664.747
org.jboss.weld.injection.ConstructorInjectionPoint 2.664.737
org.jboss.weld.injection.producer.BeanInjectionTarget 2.664.737
org.jboss.weld.injection.producer.DefaultInstanciator 2.664.731
Current Configuration
Java: Java(TM) SE Runtime Environment, 1.7.0_51-b13
JVM: Java HotSpot(TM) 64-Bit Server VM, 24.51-b03, mixed mode
Server: Server GlassFish Server Open Source Edition 4.0
Our current settings are:
-XX:+UnlockDiagnosticVMOptions
-XX:+UseParNewGC
-XX:ParallelGCThreads=4
-XX:+UseConcMarkSweepGC
-XX:MaxPermSize=1024m
-XX:NewRatio=4
-Xms4096m
-Xss256k
-Xmx8G
-javaagent:/opt/glassfish4/glassfish/lib/monitor/flashlight-agent.jar
[...]
Server Hardware
One server has 14 (including hyperthreading) CPU Cores, 32GB of ram and runs Glassfish and ElasticSearch. The Load never reaches 5 till Heap is nearly full and excessive garbage collecting kicks in. Load then rises massively.
What are we doing wrong here?
Weld, which is the implementation of CDI in Glassfish has a known memory leak in the version that ships with Glassfish 4. See: https://community.jboss.org/message/848709. You can replace the weld jar file to upgrade to version 2.0.5.Final. This will sorta fix the situation in that you will no longer see "org.jboss.weld.bean.interceptor.WeldInterceptorClassMetadata" classes grow without bound in your heap dumps.
There is another memory leak due to using tag files with parameters with CDI enabled (which is generally the case since GlassFish 4 will eagerly enable CDI by default even if you don't use it or have a beans.xml). As a workaround disable CDI or use a jsp segment include instead of a tag. See https://www.java.net/forum/topic/glassfish/glassfish/glassfish-4-memory-leak
I have a web app running on Tomcat 6.0.35, which makes use of Spring 3.1.2, Hibernate 4.1.8 and MySQL Connector 5.1.21.
I have been trying to figure out what is causing Tomcat to keep running out of memory (Perm Gen) after a few redeploys.
Note: Don't tell me to increase Tomcat's JVM memory because that will simply postpone, the problem
Specifically, I made use of the VisualVM tool, and was able to eliminate some problems, including some mysql and google threads issues. I was also able to discover and fix a problem caused by using Velocity as a singleton in the web app, and also not closing at the correct time/place some thread local variables I was having. But I still am not completely able to eliminate/figure out this Hibernate issue.
Here is what I'm doing:
Deploy my webapp from my development IDE
Open a tomcat manager window in my browser
Start VisualVM and get the HeapDump on the tomcat instance
Go the tomcat manager and redeploy my webapp
Take another HeapDump in VisualVM
My first observation is that the WebappClassLoader for the original webapp is not garbage collected.
When I scrutinize the retained objects from the second HeapDump, the class org.hibernate.internal.SessionFactoryImpl features prominently which leads me to believe that it IS NOT being destroyed/closed by Spring or something along those lines (and hence the WebappClassLoader still having a reference to it).
Has anyone encountered this problem and identified the correct fix for it?
I don't currently have an idea what could be amiss in your setup but what I know is that using Plumbr you'll most likely find the actual leak(s).