In our online cluster, we got lots of such warning:
2020-01-21 09:08:00,711 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 2328ms
No GCs detected
As it showed, there is no GC, but jvm just paused.
We noticed that the waring happens every 1 minute, and when such warning happens, cpu usage of cmf-agent increase to 80%. So we try shutdown cmf-agent, after cmf-agent is down, the waring just gone.
We have disabled Cloudera Manager jstack monitoring, what else cloudera managager agent do will cause jvm pause?
we are using CDH 5.4.14.
Edit at 2020-01-22:
/usr/java/default/bin/java -Dproc_namenode -Xmx1000m -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Dhadoop.log.dir=/var/log/hadoop-hdfs -Dhadoop.log.file=hadoop-cmf-hdfs-NAMENODE-1350.log.out -Dhadoop.home.dir=/opt/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop -Dhadoop.id.str=hdfs -Dhadoop.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/GPLEXTRAS-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.14.4-1.cdh5.14.4.p0.3/lib/hadoop/lib/native -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Xms210453397504 -Xmx210453397504 -Xmn24g -XX:SurvivorRatio=2 -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:SoftRefLRUPolicyMSPerMB=0 -verbose:GC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintAdaptiveSizePolicy -XX:+PrintReferenceGC -XX:+UseGCLogFileRotation -XX:+PrintClassHistogramAfterFullGC -XX:+PrintClassHistogramBeforeFullGC -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=200M -Xloggc:/var/log/hadoop-hdfs/nn.gc.log -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
Related
I am running a Jenkins Controller in kubernetes. I have noticed that the controller has been restarting ALOT.
kgp jkmaster-0
NAME READY STATUS RESTARTS AGE
jkmaster-0 1/1 Running 8 30m
The memory allocation to the pod is as follows
Limits:
memory: 2500M
Requests:
cpu: 300m
memory: 1G
As long as the controller is idle, I dont see any spikes occurring. But as soon as I start spawning jobs, I notice that there are spikes and each spike results in a OOMError and a restart happens
kgp jkmaster-0
NAME READY STATUS RESTARTS AGE
jkmaster-0 0/1 OOMKilled 3 3h8m
Inorder to look into this further, I would like to generate a Heap Dump. So what I am done is to add the following
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/srv/jenkins/
to JAVA_OPTS. I am expecting that the next time when the jenkins controller hits OOM, it should generate a Heapdump with under /srv/jenkins/ but there is none. Any idea if there is something I have missed ?
There is no file of the type java_pid.hprof under /srv/jenkins/ after a restart.
All JAVA_OPTS
JAVA_OPTS: -Djava.awt.headless=true -XX:InitialRAMPercentage=10.0 -XX:MaxRAMPercentage=60.0 -server -XX:NativeMemoryTracking=summary -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication \
-XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 -XX:+PrintFlagsFinal -Djenkins.install.runSetupWizard=false -Dhudson.DNSMultiCast.disabled=true \
-Dhudson.slaves.NodeProvisioner.initialDelay=5000 -Dsecurerandom.source=file:/dev/urandom \
-Xlog:gc:file=/srv/jenkins/gc-%t.log -Xlog:gc*=debug -XX:+AlwaysPreTouch -XX:+DisableExplicitGC \
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/srv/jenkins/ -Dhudson.model.ParametersAction.keepUndefinedParameters=true -Dhudson.model.DownloadService.noSignatureCheck=true
I have installed Sonar qube 6.1. I have started sonar by executing sonar.sh start and checked status with sonar.sh status. After few seconds, sonar stopping itself by writing below statements in the wrapper.log file. Please help me on this issue to get resolve.
2017.05.04 07:01:13 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube-6.1/temp
2017.05.04 07:01:13 INFO app[][o.s.p.m.JavaProcessLauncher] Launch process[es]: /usr/java/jdk1.8.0_111/jre/bin/java -Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djna.nosys=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonarqube-6.1/temp -javaagent:/usr/java/jdk1.8.0_111/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer /opt/sonarqube-6.1/temp/sq-process4451619261882420874properties
2017.05.04 07:01:23 INFO app[][o.s.p.m.Monitor] Process[es] is up
2017.05.04 07:01:23 INFO app[][o.s.p.m.JavaProcessLauncher] Launch process[web]: /usr/java/jdk1.8.0_111/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.management.enabled=false -Djruby.compile.invokedynamic=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonarqube-6.1/temp -javaagent:/usr/java/jdk1.8.0_111/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/server/*:/opt/sonarqube-6.1/lib/jdbc/h2/h2-1.3.176.jar org.sonar.server.app.WebServer /opt/sonarqube-6.1/temp/sq-process3625744255132410453properties
2017.05.04 07:01:25 INFO app[][o.s.p.m.Monitor] Process[es] is stopping
2017.05.04 07:01:26 INFO app[][o.s.p.m.Monitor] Process[es] is stopped
es is ElasticSearch, a component that indexes your data to accelerate searching. Maybe your indexes are corrupted. You can rebuild them by removing contents of ${SONARQUBE_HOME}/data/es/ directory, and retry.
After rebooting our Sonar Server running an up-to-date Amazon Linux it would not start.
2017.01.27 10:36:42 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonar/temp
2017.01.27 10:36:42 INFO app[][o.s.p.m.JavaProcessLauncher] Launch process[es]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121 0.b13.29.amzn1.x86_64/jre/bin/java -Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djna.nosys=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonar/temp -javaagent:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.29.amzn1.x86_64/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer /opt/sonar/temp/sq-process7514074371677370036properties
2017.01.27 10:36:51 INFO app[][o.s.p.m.Monitor] Process[es] is up
2017.01.27 10:36:51 INFO app[][o.s.p.m.JavaProcessLauncher] Launch process[web]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.29.amzn1.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.management.enabled=false -Djruby.compile.invokedynamic=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonar/temp -javaagent:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.29.amzn1.x86_64/jre/lib/management-agent.jar -cp ./lib/common/*:./lib/server/*:/opt/sonar/lib/jdbc/postgresql/postgresql-9.4.1209.jre7.jar org.sonar.server.app.WebServer /opt/sonar/temp/sq-process6553321235832962376properties
2017.01.27 10:36:54 INFO app[][o.s.p.m.Monitor] Process[es] is stopping
2017.01.27 10:36:54 ERROR app[][o.s.p.m.Monitor] Process[web] failed to start
2017.01.27 10:36:54 INFO app[][o.s.p.m.Monitor] Process[es] is stopped
<-- Wrapper Stopped
The above indicates you need to read /var/log/sonar/web.log
The postgres jar above /opt/sonar/lib/jdbc/postgresql/postgresql-9.4.1209.jre7.jar is a bad smell, so on a hunch I downloaded the latest version,
which fixed the problem.
I imagine this will be fixed by the Sonar team pretty soon but this posted here in the hope that it helps.
Upgraded from 5.6 to 6.1. Sonar.properties are, and they match what was there before:
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
sonar.jdbc.url=jdbc:mysql://10.120.8.27:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance
sonar.web.javaOpts=-Xmx768m -XX:MaxPermSize=160m -XX:+HeapDumpOnOutOfMemoryError
sonar.web.host=10.120.8.27
sonar.web.context=/sonar
sonar.web.port=9000
SonarQube starts and is running but it's not listening on port 9000. OS is CentOS 6.6, SElinux disabled. I can find no errors in the logs, indeed sonar.log has no entries at all. What have I missed?
ps -ef|grep sonar
sonar 1325 1 0 12:06 ? 00:00:00
/opt/sonar/bin/linux-x86-64/./wrapper
/opt/sonar/bin/linux-x86-64/../../conf/wrapper.conf
wrapper.syslog.ident=SonarQube
wrapper.pidfile=/opt/sonar/bin/linux-x86-64/./SonarQube.pid
wrapper.daemonize=TRUE
sonar 1327 1325 2 12:06 ? 00:00:25 java
-Djava.awt.headless=true -Xms3m -Xmx3m -Djava.library.path=./lib -classpath
../../lib/jsw/wrapper-3.2.3.jar:../../lib/sonar-application-6.1.jar
-Dwrapper.key=byv_fqDBq6Ok__mi -Dwrapper.port=32000
-Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999
-Dwrapper.debug=TRUE -Dwrapper.pid=1325 -Dwrapper.version=3.2.3
-Dwrapper.native_library=wrapper -Dwrapper.service=TRUE
-Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1
org.tanukisoftware.wrapper.WrapperSimpleApp org.sonar.application.App
sonar 1545 1327 99 12:06 ? 00:24:20
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/jre/bin/java
-Djava.awt.headless=true -Xmx1G -Xms256m -Xss256k -Djna.nosys=true
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/opt/sonar/temp
-javaagent:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/jre/lib/management-agent.jar
-cp ./lib/common/*:./lib/search/* org.sonar.search.SearchServer
/opt/sonar/temp/sq-process7944098912777335007properties
Was a permissions problem on the logs. From there log tracing lead to other, but clear problems.
I ran a simple CLion project with only two directories which each contain a single file. However, I was getting this message “Out of Memory”. I increased the Xmx from 2GB to 4GB. I believe this is more than enough. How can I fix this problem?
This is the settings in the clion64.exe.vmoptions file:
-Xss2m
-Xms256m
-Xmx4000m
-XX:MaxPermSize=350m
-XX:ReservedCodeCacheSize=96m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
try there settings:
-Xss2m
-Xms256m
-Xmx2000m
-XX:NewSize=128m
-XX:MaxNewSize=128m
-XX:MaxPermSize=350m
-XX:ReservedCodeCacheSize=96m
-XX:+UseConcMarkSweepGC
-XX:SoftRefLRUPolicyMSPerMB=50
-ea
-Dsun.io.useCanonCaches=false
-Djava.net.preferIPv4Stack=true
refer to this page for more info:
link