JenkinsX - Build fail due to Java Heap space - jenkins-x

I am working on creating CI/CD pipeline for Spring Boot application on GKE using JenkinsX. As soon as I push the code to the master branch, build gets triggered but the build fails due to insufficient Java heap space.
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.2.2:war (default-war) on project location-finder-api: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.2.2:war (default-war) on project location-finder-api: Error assembling WAR: Problem creating war: Execution exception
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.needNewBuffer (ByteArrayOutputStream.java:153)
at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.write (ByteArrayOutputStream.java:192)
To resolve I tried to set the JVM argument in the Docekrfile as
CMD ["java", "-Xmx1024m","-jar", "app.jar"]
But it did not work. This is what I see when the build starts
+ mvn -e clean deploy -Pprod
Picked up _JAVA_OPTIONS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx192m
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
Is there any way that I can set this heap option on my own?

it looks like maven is running out of memory so you need more memory in your build pod (not the Dockerfile for your app).
as a quick test you can edit the pod template inside the Jenkins UI: jx console then Manage Jenkins -> Configure System then find the jenkins-maven pod template in the UI and edit the _JAVA_OPTIONS environment variable from this value: https://github.com/jenkins-x/jenkins-x-platform/blob/master/jenkins-x-platform/values.yaml#L907 - try change -Xmx512m to something bigger like -Xmx912m
Once you've found a value that works for your projects you can make the change permanent of restarts of Jenkins by adding this to your myvalues.yaml - something like this...
# myvalues.yaml
jenkins:
Agent:
PodTemplates:
Maven:
Name: maven
Label: jenkins-maven
EnvVars:
_JAVA_OPTIONS: '-XX:+UnlockExperimentalVMOptions -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xms10m -Xmx912m'
see the docs on creating/configuring builders

Related

native build failure: classes specified in --initialize-at-run-time still say that they are requested to be initialized at run time

I'm new to Quarkus and trying to build a native image using "mvn package -Pnative". I received the error "Error: Classes that should be initialized at run time got initialized during image building:" so in the pom.xml I added an additional build arg --initialize-at-run-time with a \, separated list for these classes. When I run the mvn command again, all of these classes still say that they need to be initialized at run time.
I tried to build the native image, got the Error that several classes need to be initialized at run time, and so tried to build again after adding --initialize-at-run-time in the additional build args with the specified classes. I expected that if a class is added to --initialize-at-run-time, it should not give me the error that the same class needs to be initialized at run time when running the mvn command, and the build should succeed.
Process and details:
I entered into the Intellij terminal : "mvn package -Pnative"
After finishing tests, there is the following INFO line :
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildRunner] /Library/Java/JavaVirtualMachines/graalvm-ce-java11-22.3.0/Contents/Home/bin/native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dlogging.initial-configurator.min-level=500 -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=3 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true -J-Duser.language=en -J-Duser.country=US -J-Dfile.encoding=UTF-8 --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.ResourcesFeature,io.quarkus.runtime.graal.DisableLoggingFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+CollectImageBuildStatistics -H:ImageBuildStatisticsFile=[PROJECT-NAME]-1.0.0-SNAPSHOT-runner-timing-stats.json -H:BuildOutputJSONFile=[PROJECT-NAME]-1.0.0-SNAPSHOT-runner-build-output-stats.json --initialize-at-run-time=io.netty.buffer.UnpooledDirectByteBuf,io.netty.handler.ssl.ReferenceCountedOpenSslContext,io.netty.buffer.UnpooledHeapByteBuf,io.netty.buffer.ByteBufUtil,io.netty.buffer.AbstractReferenceCountedByteBuf,io.netty.channel.kqueue.KQueue,io.netty.buffer.UnpooledByteBufAllocator\$InstrumentedUnpooledUnsafeDirectByteBuf,io.netty.handler.ssl.OpenSslContext,io.netty.util.concurrent.ScheduledFutureTask,io.netty.buffer.AbstractPooledDerivedByteBuf,io.netty.buffer.PooledByteBufAllocator,io.netty.handler.ssl.ReferenceCountedOpenSslClientContext,io.netty.buffer.UnpooledUnsafeDirectByteBuf,io.netty.channel.DefaultFileRegion,io.netty.handler.ssl.OpenSslClientContext,io.netty.util.concurrent.GlobalEventExecutor,io.netty.channel.epoll.Epoll,io.netty.util.AbstractReferenceCounted,io.netty.buffer.ByteBufAllocator,io.netty.buffer.PooledSlicedByteBuf,io.netty.buffer.PooledByteBuf,io.netty.buffer.PooledUnsafeDirectByteBuf -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy\$BySpaceAndTime -H:+AllowFoldMethods -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+ReportExceptionStackTraces -H:-AddAllCharsets --enable-url-protocols=http,https -H:-UseServiceLoaderFeature -H:+StackTrace -J--add-exports=org.graalvm.sdk/org.graalvm.nativeimage.impl=ALL-UNNAMED -J--add-exports=org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk=ALL-UNNAMED [PROJECT-NAME]-1.0.0-SNAPSHOT-runner -jar [PROJECT-NAME]-1.0.0-SNAPSHOT-runner.jar
it runs through 1/7 initializing and 2/7 performing analysis until hitting errors like:
Error: Classes that should be initialized at run time got initialized during image building: io.netty.buffer.AbstractReferenceCountedByteBuf the class was requested to be initialized at run time ....... in 'file:///path/to/project/target/[PROJECT-NAME]-1.0.0-SNAPSHOT-native-image-source-jar/lib/io.netty.netty-buffer-4.1.79.Final.jar'
the end of the errors has the following line that contains all of the same classes that I had specified:
To see how the classes got initialized, use --trace-class-initialization=io.netty.channel.kqueue.KQueue,io.netty.handler.ssl.OpenSslContext,io.netty.buffer.PooledUnsafeDirectByteBuf,io.netty.buffer.UnpooledHeapByteBuf,io.netty.channel.DefaultFileRegion,io.netty.buffer.PooledSlicedByteBuf,io.netty.buffer.PooledByteBuf,io.netty.handler.ssl.ReferenceCountedOpenSslClientContext,io.netty.util.AbstractReferenceCounted,io.netty.handler.ssl.OpenSslClientContext,io.netty.buffer.AbstractPooledDerivedByteBuf,io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf,io.netty.buffer.ByteBufUtil,io.netty.channel.epoll.Epoll,io.netty.util.concurrent.GlobalEventExecutor,io.netty.buffer.UnpooledUnsafeDirectByteBuf,io.netty.buffer.UnpooledDirectByteBuf,io.netty.buffer.AbstractReferenceCountedByteBuf,io.netty.handler.ssl.ReferenceCountedOpenSslContext,io.netty.buffer.PooledByteBufAllocator,io.netty.buffer.ByteBufAllocator,io.netty.util.concurrent.ScheduledFutureTask
the different causes after adding trace-class-initialization to the pom were:
com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdLoopNativeDetector
reactor.netty.http.client.HttpClientSecure
com.azure.cosmos.implementation.directconnectivity.rntbd.RntbdServiceEndpoint$RntbdEndpointMonitoringProvider
reactor.netty.resources.DefaultLoopNativeDetector
io.netty.handler.ssl.OpenSsl
I also tried adding these to the --initialize-at-run-time but that did not resolve it
the message after build failure:
[ERROR] Failed to execute goal io.quarkus.platform:quarkus-maven-plugin:2.13.1.Final:build on project: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
I'm unable to provide the project as it contains sensitive and internal info but I added the following to the native profile in my pom.xml to try and resolve the error:
<quarkus.native.additional-build-args>--initialize-at-run-time=io.netty.buffer.UnpooledDirectByteBuf\,io.netty.handler.ssl.ReferenceCountedOpenSslContext\,io.netty.buffer.UnpooledHeapByteBuf\,io.netty.buffer.ByteBufUtil\,io.netty.buffer.AbstractReferenceCountedByteBuf\,io.netty.channel.kqueue.KQueue\,io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeDirectByteBuf\,io.netty.handler.ssl.OpenSslContext\,io.netty.util.concurrent.ScheduledFutureTask\,io.netty.buffer.AbstractPooledDerivedByteBuf\,io.netty.buffer.PooledByteBufAllocator\,io.netty.handler.ssl.ReferenceCountedOpenSslClientContext\,io.netty.buffer.UnpooledUnsafeDirectByteBuf\,io.netty.channel.DefaultFileRegion\,io.netty.handler.ssl.OpenSslClientContext\,io.netty.util.concurrent.GlobalEventExecutor\,io.netty.channel.epoll.Epoll\,io.netty.util.AbstractReferenceCounted\,io.netty.buffer.ByteBufAllocator\,io.netty.buffer.PooledSlicedByteBuf\,io.netty.buffer.PooledByteBuf\,io.netty.buffer.PooledUnsafeDirectByteBuf</quarkus.native.additional-build-args>
I'm running on a Mac with M1/aarch64, Monterey 12.6. java -version output is:
openjdk version "11.0.17" 2022-10-18 OpenJDK Runtime Environment GraalVM CE 22.3.0 (build 11.0.17+8-jvmci-22.3-b08) OpenJDK 64-Bit Server VM GraalVM CE 22.3.0 (build 11.0.17+8-jvmci-22.3-b08, mixed mode)
and echo $JAVA_HOME is /Library/Java/JavaVirtualMachines/graalvm-ce-java11-22.3.0/Contents/Home". GraalVM is version 22.3.0, Quarkus-maven-plugin is version 2.13.1.Final, and I use maven 3.8.6.

Jenkins maven projet build failed on fork during test

I'm building maven project on Jenkins (version 2.263.4, not the last but not so far).
It's a git project with pipeline configuration and webhook from bitbucket when push is done on the project.
So when a developper create a branch and another one a pull request, i have 2 build in parallel. Domain test running in about 45min.
And sometimes(not always), i have this error :
Command was /bin/sh -c cd /jenkins/workspace/a-hub_test-plantage-jenkins_PR-1/server/project-core/project-core-domain && /jenkins/tools/hudson.model.JDK/jdk11/jdk-11.0.2+9/bin/java -Xmx1500M -jar /jenkins/workspace/a-hub_test-plantage-jenkins_PR-1/server/project-core/project-core-domain/target/surefire/surefirebooter15630998754365374145.jar /jenkins/workspace/a-hub_test-plantage-jenkins_PR-1/server/project-core/project-core-domain/target/surefire 2021-05-04T12-39-54_622-jvmRun1 surefire5846464826560071036tmp surefire_13691263758446667285tmp
Error occurred in starting fork, check output in log
14:46:13 [ERROR] Process Exit Code: 1
14:46:13 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
14:46:13 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
14:46:13 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
If i re run the buildin error, it work.
I'm using maven-surefire 2.22.2.
I try to configure it following many thread on stackoverfow, but nothgin work. I try with forkMode none or something else, it work but i loose coverage because jaccoco need fork to work. So i'm stuck.
What can i do to help me to debug this problem?
I don't understand why build failed when another one is building because for me Jenkins work on different working folder, no? so why the problem appear when i'have multiple build in parallel?
thanks for help
I find an hs_err_pid.log in jenkins near source code and i see out of memory! i see that my test was launch with -Xmx1500Mo and problem occured when i launched multiple branch/pr, so jenkins use all of the memory and failed all build during fork...i decrease -Xmx and all seems to work.

Connecting to the Gradle build

I am working with Liferay 7.2. I am using Liferay Developer Studio and Gradle as a build tool.
While creating a new module project after creating of 1-2 new, getting error message in IDE's workspace .log file and new module project is not creating.
!MESSAGE Synchronize project liferay-workspace failed due to an error connecting to the Gradle build. !STACK 0 org.gradle.tooling.GradleConnectionException: Could not run build action using Gradle distribution 'https://services.gradle.org/distributions/gradle-4.10.2-bin.zip'.
In C:\Users\nirav.prajapati.gradle\daemon\4.10.2\daemon-11084.out file following error shows.
Problem in daemon expiration check java.lang.OutOfMemoryError: GC
overhead limit exceeded FAILURE: Build failed with an exception.
* What went wrong: GC overhead limit exceeded
* Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Problem in daemon expiration check java.lang.OutOfMemoryError: Java
heap space Failed to execute
org.gradle.launcher.daemon.server.Daemon$DaemonExpirationPeriodicCheck#7f47f3a7.
java.lang.OutOfMemoryError: Java heap space FAILURE: Build failed with
an exception.
* What went wrong: Failed to notify build listener. Java heap space
I have increased JVM Heap size by double click on IDE's Server.
Even newly created project is not showing into Gradle Task. In Project Explorer it's showing like incomplete project means folder structure are different then exists. Moreover, .projects and .settings file is also not creating in that.
Multiple time I have make fresh configuration like, remove Liferay developer studio, workspace and .gradle folder from C:\User\
After getting this error, I have restarted system and start again liferay developer studio, but Synchronize Gradle Project with Workspace progress is taking more than 45 min. to complete the process.
Please help me to solve it.
Please these properties in gradle.properties file.
org.gradle.parallel=true
org.gradle.jvmargs=-Xmx3g -XX:MaxPermSize=500m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8

Gradle sonarqube plugin - how to set memory for findbugs?

I have a project module with 30K classes.
After migrating sonar analysis from ant script to gradle plugin I have OOM error with output like this:
13:10:36 Out of memory
13:10:36 Total memory: 954M
13:10:36 free memory: 119M
13:10:52 Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
13:10:52 at org.sonar.plugins.findbugs.FindbugsExecutor.execute(FindbugsExecutor.java:163)
13:10:52 ... 109 more
13:10:52 Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
13:10:52 at
We've run ant script with the following parameters "-Xmx3800m -XX:ReservedCodeCacheSize=128m"
How can I set the same parameters for sonarqube gradle plugin?
I've tried setting the following env variable before calling gradle
GRADLE_OPTS=-Xmx3800m -XX:ReservedCodeCacheSize=128m
It's applied correctly, but findbugs still failing and prints "Total memory: 954M"
Also I've tried adding the following properties to reduce memory consumption, but without any luck
property 'sonar.skipPackageDesign', 'true'
property 'sonar.skipDesign', 'true'
Gradle version is 3.5
Sonarqube plugin version is 2.5
jdk version is 8u131
I realized what was the problem.
Setting GRADLE_OPTS was correct solution, however I also used gradle daemon, so those options were ignored for daemon.
I ended up disabling daemon by adding -Dorg.gradle.daemon=false to GRADLE_OPTS and it worked.
Check build.gradle file and set maxHeapSize as given below,
tasks.withType(FindBugs) {
maxHeapSize = "1000m"
}
You can change heap size.

Jenkins fails during parsing poms

I have problems with one job on my Jenkins server.
It fails during pom parse with this message:
Parsing POMs
Modules changed, recalculating dependency graph
[workspace] $ java -Xmx512m -XX:MaxPermSize=256m -Djava.awt.headless=true -cp /opt/edb/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-agent-1.2.jar:/opt/apache/maven3/boot/plexus-classworlds-2.4.jar org.jvnet.hudson.maven3.agent.Maven3Main /opt/apache/maven3 /var/lib/tomcat6/webapps/jenkins/WEB-INF/lib/remoting-2.17.jar /opt/edb/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-1.2.jar 55430
<===[JENKINS REMOTING CAPACITY]===>channel started
ERROR: Failed to parse POMs
java.io.IOException: Remote call on Channel to Maven [java, -Xmx512m, -XX:MaxPermSize=256m, -Djava.awt.headless=true, -cp, /opt/edb/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-agent-1.2.jar:/opt/apache/maven3/boot/plexus-classworlds-2.4.jar, org.jvnet.hudson.maven3.agent.Maven3Main, /opt/apache/maven3, /var/lib/tomcat6/webapps/jenkins/WEB-INF/lib/remoting-2.17.jar, /opt/edb/jenkins/plugins/maven-plugin/WEB-INF/lib/maven3-interceptor-1.2.jar, 55430] failed
at hudson.remoting.Channel.call(Channel.java:673)
at hudson.maven.ProcessCache$MavenProcess.<init>(ProcessCache.java:112)
at hudson.maven.ProcessCache.get(ProcessCache.java:231)
at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:704)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
at hudson.model.Run.execute(Run.java:1516)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:477)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
Caused by: java.lang.Error: Failed to deserialize the Callable object.
Caused by: java.lang.IndexOutOfBoundsException: Index: 1997078527, Size: 0
I have tried creating a new build, no luck
Building locally works fine
All other similar jobs work fine
I'm running letest jenkins (1.489)
Any ideas?
Hard disk is full.
Try to restart.
Maybe the workspace is corrupted and on your pom.xml is not readable. Have to try to clean the workspace for this specific job ?
This issue was fixed for me by setting the correct version Java version in the PATH (as required in the pom.xml) - See https://issues.jenkins-ci.org/browse/JENKINS-5519

Resources