1. Condition

1. Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
2. Jira
   JVM_MINIMUM_MEMORY=”2g”
   JVM_MAXIMUM_MEMORY=”3g”
3. Confluence
   
   CATALINA_OPTS="-Xms2g -Xmx3g -XX:+UseG1GC ${CATALINA_OPTS}"

   1	CATALINA_OPTS="-Xms2g -Xmx3g -XX:+UseG1GC ${CATALINA_OPTS}"

4. BitBucket
   
   JVM_MINIMUM_MEMORY=2g JVM_MAXIMUM_MEMORY=2g

   1 2	JVM_MINIMUM_MEMORY=2g JVM_MAXIMUM_MEMORY=2g

   
   and elasticsearch:
   
   -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss320k

   1	-Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss320k

5. Artifactory
   
   export JAVA_OPTIONS="-server -Xms512m -Xmx1g -Xss256k -XX:+UseG1GC -XX:OnOutOfMemoryError=\"kill -9 %p\""

   1	export JAVA_OPTIONS="-server -Xms512m -Xmx1g -Xss256k -XX:+UseG1GC -XX:OnOutOfMemoryError=\"kill -9 %p\""

6. Jenkins
   
   JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx1024m"

   1	JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xmx1024m"

2. Problem

3. Analyzing

3-1. Try 1

After reading this Bitbucket Server is reaching resource limits, I tried to set the JVM_MAXIMUM_MEMORY to 3gb for Bitbucket. But I can not start Bitbucket, with such error: cannot allocate 3xxxxxxk.

3-2. Try 2

I guess there is one system background process had like to kill bitbucket process unexpectedly, and I found this post:

Bitbucket Server Process Dies Unexpectedly Due to Linux OOM-Killer

The Symptom is 100% the same as mine:

• Bitbucket Server is installed on a Linux host.
•  The entire Bitbucket Server process suddenly terminates without warning. That is to say, the process ID (pid) is gone.
• The browser shows a generic “cannot connect” or similar error, indicating that it is not able to reach the webpage
• Nothing out of the ordinary appears in the Bitbucket Server application logs ($BITBUCKET_HOME/logs/atlassian-bitbucket.log), since the application was terminated without properly shutting down
• Similarly, Tomcat logs (<Bitbucket Server installation directory>/logs/catalina.out) also do not show errors

The Diagnosis from this article said that:

 Linux OOM-Killer is a feature on some Linux installations that will sacrifice processes to free up memory if the operating system experiences memory exhaustion for its own operations. Please note that this is different from Bitbucket Server running out of memory. In this case, the OS itself is in danger of running out of memory and thus starts terminating processes to avoid it.
On the host machine, look in the /var/log/ directory for the syslog or messages.

3-3. Try 3

An idea flashed through my mind, I can try running Bitbucket with lower memory.

And miracle comes, the Bitbucket is running stable. WTF?

Reason unknown. 2G is a huge size for a 20Gb memory free host??

3-5. Try 4

4. Conclusion