Increase the building heap-size for Jenkins jobs
In Jenkins configuration the following configs have been set:
Maven Project Configuration->Global MAVEN_OPTS
1 |
-Xmx8192m -XX:MaxPermSize=1024m |
Global properties->Environment variables
1 2 3 4 |
Name: MAVEN_OPTS Value: -Xmx8192m -XX:MaxPermSize=1024m -Djava.io.tmpdir=/clusters/shared_workspace/tmp |
We have a Jenkins job to build a huge Java project, and it has often such OutOfMemoryError during the building process.
1 2 3 4 5 6 7 8 9 10 11 12 |
<span class="pipeline-node-56">Exception in thread "Thread-177" java.lang.OutOfMemoryError: GC overhead limit exceeded at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:98) at com.codahale.metrics.ExponentiallyDecayingReservoir.update(ExponentiallyDecayingReservoir.java:84) at com.codahale.metrics.Histogram.update(Histogram.java:41) at com.codahale.metrics.Timer.update(Timer.java:185) at com.codahale.metrics.Timer.update(Timer.java:89) at io.micrometer.core.instrument.dropwizard.DropwizardTimer.recordNonNegative(DropwizardTimer.java:43) at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:171) at io.micrometer.core.instrument.AbstractTimer.recordValueWithExpectedInterval(AbstractTimer.java:130) at io.micrometer.core.instrument.AbstractTimer.lambda$initPauseDetector$1(AbstractTimer.java:116) at io.micrometer.core.instrument.AbstractTimer$$Lambda$384/1481847479.handlePauseEvent(Unknown Source) at org.LatencyUtils.PauseDetector$PauseDetectorThread.run(PauseDetector.java:107)</span> |
The heap memory logs looks like this:
1 2 3 |
<span class="pipeline-node-56">processors=16, physical.memory.total=62.7G, physical.memory.free=10.6G, swap.space.total=4.0G, swap.space.free=3.8G, heap.memory.used=864.7M, heap.memory.free=103.8M, heap.memory.total=968.5M, heap.memory.max=968.5M, heap.memory.used/total=89.17%, heap.memory.used/max=89.17%, minor.gc.count=269, minor.gc.time=3852ms, major.gc.count=418, major.gc.time=506094ms,</span> |
The running Jenkins was built with a docker cluster, one master with two workers.
1. 1. Set docker start JAVA_OPTS
The first try is to set the initialized Java options when starting the container:
1 2 3 4 5 6 7 8 9 10 |
master: image: jenkins/jenkins:lts-jdk11 user: '88396' ports: - "8080" build: context: . dockerfile: Dockerfile-jenkins-master environment: - "JAVA_OPTS=-Djava.awt.headless=true -Xmx12g -Xms12g -Djava.net.preferIPv4Stack=true -Duser.home=/jenkins_home -Duser.timezone=Europe/Berlin -Dorg.apache.commons.jelly.tags.fmt.timeZone=Europe/Berlin" |
The VM has 64GB of memory and it should allow to run three 12GB memory containers and will not face memory issues. After testing, the situation gets better, but the occurrence of failed building jobs is changing from often to not so often.
Then I try to set an even larger Xmx from 12g to 16g. The situation gets worse, the Jenkins container was killed by the system sometimes. VM allocates up to 25% of your RAM per JVM running on your machine, similar to the docker environment. It seems that 16GB crosses the red line.
2. 2. Set Maven Memory Limits
2-1. 2.1 Set in Project
Following this http://tutorials.jenkov.com/maven/maven-memory-limits.html
I add a jvm.config in my project like the following structure:
- myproject
- .mvn
- jvm.config
- .mvn
Inside the jvm.config file you need the following settings to be able to control the memory limits of Maven during a Maven build:
1 |
-Xmx6144m -Xms2048m |
It did change the JVM option when the job is running, but it will be overwritten by the Jenkins configuration.
This could help you to build projects locally with larger memory, but not helpful in Jenkins.
2-2. 2.2 Set in Maven’s mvn.sh
Reference: https://techtractall.wordpress.com/2017/02/22/how-to-increase-java-heap-space-on-maven-and-ant/
Tested, it does not help.
3. 3. Set Project POM maven-compiler
After several testing I noticed that in logs, the heap memory never changes, it keeps with less than 1GB.
1 2 |
heap.memory.used=864.7M, heap.memory.free=103.8M, heap.memory.total=968.5M, heap.memory.max=968.5M, heap.memory.used/total=89.17%, heap.memory.used/max=89.17% |
How can increase the heap memory size when compiling the project on Jenkins?
Ultimately I found this: https://stackoverflow.com/questions/12498738/maven-out-of-memory-build-failure
1 2 3 4 5 6 7 8 9 |
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <fork>true</fork> <meminitial>1024m</meminitial> <maxmem>2024m</maxmem> </configuration> </plugin> |
After testing, you guess what? The heap size has been changed!
1 2 3 |
processors=16, physical.memory.total=62.6G, physical.memory.free=1.6G, swap.space.total=4.0G, swap.space.free=1.3G, heap.memory.used=<span style="color: #ff0000;"><strong>1.5G</strong></span>, heap.memory.free=<span style="color: #ff0000;"><strong>4.3G</strong></span>, heap.memory.total=<strong><span style="color: #ff0000;">5.7G</span></strong>, heap.memory.max=13.9G, heap.memory.used/total=25.42%, heap.memory.used/max=10.45%, |
There is no more OutOfMemoryError issue!
More detail: https://maven.apache.org/plugins/maven-compiler-plugin/examples/compile-with-memory-enhancements.html
4. 4. How to set JVM for Jenkins Node (Slave)
If you face the OutOfMemoryError issue on the node, add this property in node configuration.