Java Heap Space Error | OutOfMemory while deploying EAR in Websphere - websphere

Question

I am getting OutOfMemory error while deploying EAR (Size : 230MB) file in Websphere server.
Sometime deployment is getting success after increasing the heap size.
I have analyzed the heap dump and found leak suspects but not sure how to proceed here after.
Leak suspect : 217,295,824 bytes (87.23 %) of Java heap is used by 105
instances of java/util/WeakHashMap$Entry
Contains 3 instances of the following leak suspects:
- array of java/lang/Object holding 16,235,440 bytes at 0x6a696c8
- array of java/lang/Object holding 101,373,968 bytes at 0x1125c240
- array of java/lang/Object holding 13,602,688 bytes at 0x5290818
<\n> Total size : 217,295,824 bytes
Size : 1,040 bytes
Name : array of java/util/WeakHashMap$Entry
Number of children : 105
Number of parents : 1
Owner address : 0x2e41fd0
Owner object : java/util/WeakHashMap
Address : 0xb4c2dc0
First single ancestor : org/eclipse/jst/j2ee/internal/archive/JavaEEArchiveUtilities at 0xb4c2dc0
and getting below error in WAS logs
[main] INFO deploylib - Installing application... ADMA5016I: Installation of Kijkglas-ear-1905.01.35 started. ADMA5058I: Application and module versions are validated with versions of deployment targets. ADMA5018I: The EJBDeploy program is running on file /tmp/app6232412827642995266.ear. Starting workbench. EJB Deploy configuration directory: /var/was/profiles/AdminAgent01/ejbdeploy/configuration/ framework search path: /opt/IBM/WebSphere/8.5/deploytool/itp/plugins build:RADWEJB95-I20150829_0214 Creating the project. JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2019/06/07 10:42:59 - please wait. JVMDUMP032I JVM requested System dump using '/var/was/profiles/AdminAgent01/core.20190610.104259.30244.0001.dmp' in response to an event JVMDUMP010I System dump written to /var/was/profiles/AdminAgent01/core.20190610.104259.30244.0001.dmp JVMDUMP032I JVM requested Heap dump using '/var/was/profiles/AdminAgent01/heapdump.20190610.104259.30244.0002.phd' in response to an event JVMDUMP010I Heap dump written to /var/was/profiles/AdminAgent01/heapdump.20190610.104259.30244.0002.phd JVMDUMP032I JVM requested Java dump using '/var/was/profiles/AdminAgent01/javacore.20190610.104259.30244.0003.txt' in response to an event JVMDUMP010I Java dump written to /var/was/profiles/AdminAgent01/javacore.20190610.104259.30244.0003.txt JVMDUMP032I JVM requested Snap dump using '/var/was/profiles/AdminAgent01/Snap.20190610.104259.30244.0004.trc' in response to an event JVMDUMP010I Snap dump written to /var/was/profiles/AdminAgent01/Snap.20190610.104259.30244.0004.trc JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". An unexpected exception was thrown. Halting execution. Shutting down workbench. Error executing deployment: java.lang.OutOfMemoryError. Error is Java heap space. java.lang.OutOfMemoryError: Java heap space at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.<init>(Throwable.java:67) at java.lang.Throwable.<init>(Throwable.java:78) at java.lang.Error.<init>(Error.java:82) at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:64) at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69) at java.lang.String.<init>(String.java:207) at java.util.jar.Attributes.read(Attributes.java:424) at java.util.jar.Manifest.read(Manifest.java:264) at java.util.jar.Manifest.<init>(Manifest.java:82) at java.util.jar.JarFile.getManifestFromReference(JarFile.java:200) at java.util.jar.JarFile.getManifest(JarFile.java:182) at sun.net.www.protocol.jar.URLJarFile.isSuperMan(URLJarFile.java:187) at sun.net.www.protocol.jar.URLJarFile.getManifest(URLJarFile.java:155) at java.util.jar.JarFile.maybeInstantiateVerifier(JarFile.java:387) at java.util.jar.JarFile.getInputStream(JarFile.java:488) at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:178) at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source) at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source) at org.apache.xerces.impl.xs.opti.SchemaParsingConfig.parse(Unknown Source) at org.apache.xerces.impl.xs.opti.SchemaParsingConfig.parse(Unknown Source) at org.apache.xerces.impl.xs.opti.SchemaDOMParser.parse(Unknown Source) at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument(Unknown Source) at org.apache.xerces.impl.xs.traversers.XSDHandler.resolveSchema(Unknown Source) at org.apache.xerces.impl.xs.traversers.XSDHandler.constructTrees(Unknown Source) at org.apache.xerces.impl.xs.traversers.XSDHandler.constructTrees(Unknown Source) at org.apache.xerces.impl.xs.traversers.XSDHandler.parseSchema(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaLoader.loadSchema(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator.findSchemaGrammar(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source) at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown Source) EJBDeploy level: #build# ADMA5008E: The EJBDeploy program failed on file /tmp/app6232412827642995266.ear. Exception: com.ibm.etools.ejbdeploy.EJBDeploymentException: Error executing EJBDeploy ADMA0063E: An error occurred during Enterprise JavaBeans (EJB) deployment. Exception: com.ibm.etools.ejbdeploy.EJBDeploymentException: Error executing EJBDeploy ADMA5011I: The cleanup of the temp directory for application Kijkglas-ear-1905.01.35 is complete. ADMA5014E: The installation of application Kijkglas-ear-1905.01.35 failed. 2019-06-10 10:43:05,625
[main] FATAL deploylib - Jython Exception in deploy.py : 2019-06-10 10:43:05,630
[main] FATAL deploylib - Traceback (most recent call last): 2019-06-10 10:43:05,630
[main] FATAL deploylib - File "/opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/deploylib/cfgfiles/gMyAppWA-assembled.cfg", line 606, in ? application.installApplication() 2019-06-10 10:43:05,630
[main] FATAL deploylib - File "<string>", line 779, in installApplication 2019-06-10 10:43:05,630
[main] FATAL deploylib - com.ibm.ws.scripting.ScriptingException: com.ibm.ws.scripting.ScriptingException: WASX7132E: Application install for /opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/Kijkglas-ear-1905.01.35.ear failed: see previous messages for details. [2019-06-10 10:43:05] [/opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/deploylib/deploy.ksh] [ERROR] Command /var/was/profiles/AppSrv01/bin/wsadmin.sh -javaoption -Duser.timezone=CET -f deploy.py /opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/deploylib/cfgfiles/gMyAppWA-assembled.cfg /opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/deploylib/cfgfiles/gMyAppWA.TST failed. [2019-06-10 10:43:05] [/opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/deploylib/deploy.ksh] [INFO ] See also deploy.log and wsadmin.log in deploylib-8.1.4 directory.
See /opt/Nolio/work/WAS/log/gMyAppWA/all/stdout.log.2019-06-10_10_37_57_285 and /opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/deploylib/deploy.log for more information
/gMyAppWA.TST failed. [2019-06-10 10:43:05] [/opt/Nolio/work/WAS/gMyAppWA/all/1905.01.35/deploylib/deploy.ksh] [INFO ] See also deploy.log and wsadmin.log in deploylib-8.1.4 directory.
Is there any rouge process or something blocking in background ?

Answer 1

You didn't write your version, topology (single, network deployment), nor way you deploy your app (console, wsadmin ohter).
As you can see in the log, there is OutOfMemoryError during EJB deploy call.
You need to increase memory for ejb deploy you can either set it file or in OS level. Check this post Getting OutofMemory condition while deploying a large application in WebSphere Application Server
1) Set it in install-root/deploytool/itp/EJBDeploy.sh file
EJBDEPLOY_JVM_HEAP="-Xms1024 -Xmx1024" at the beginning of the
ejbdeploy.sh file.
2) Set it in operating System Environment.
Set EJBDEPLOY_JVM_HEAP= '-Xms1024 -Xmx1024' in OS environment
variable.
I'd also increase memory for your admin server in AdminAgent01 profile, as it looks like you are using admin agent.

Answer 2

Recommend not to update the ejbdeploy.sh. When update the WebSphere to a new fixpack, the ejbdeploy.sh will be restored.
Increasing heap size through admin console
login to admin console
go to "System administration" > "Deployment manager" > "Configuration" tab > "Server Infrastructure" section on the right > "Java and Process Management" > "Process definition"
"Additional Properties" section on the right > "Environment Entries"
"New" entry by providing the Name EJBDEPLOY_JVM_HEAP and value "-Xms256m -Xmx1024m"
Save and synchronize
restart the DM server