How to do JVM performance analysis in docker container?
1. Prerequisite
Java application with docker based on JDK 8 and alpine.
This app is a Jhipster application.
Dockerfile:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
FROM openjdk:8-jre-alpine ENV SPRING_OUTPUT_ANSI_ENABLED=ALWAYS \ JHIPSTER_SLEEP=0 \ JAVA_OPTS="" # Add a jhipster user to run our application so that it doesn't need to run as root RUN adduser -D -s /bin/sh jhipster WORKDIR /home/jhipster ADD entrypoint.sh entrypoint.sh RUN chmod 755 entrypoint.sh && chown jhipster:jhipster entrypoint.sh USER jhipster ENTRYPOINT ["./entrypoint.sh"] EXPOSE 18082 5701/udp ADD *.jar app.jar |
entrypoint.sh
1 2 3 4 |
#!/bin/sh echo "The application will start in ${JHIPSTER_SLEEP}s..." && sleep ${JHIPSTER_SLEEP} exec java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=5005,suspend=n -jar "${HOME}/app.jar" "$@" |
2. Precheck
Enter container to check the process. Of course, we have only one process from the app.
Enter container with root role.
1 |
> docker exec -it -u 0 container_id sh |
Java version:
1 2 3 4 5 |
~ $ java -version Picked up _JAVA_OPTIONS: -Xmx8192m -Xms4096m openjdk version "1.8.0_111-internal" OpenJDK Runtime Environment (build 1.8.0_111-internal-alpine-r0-b14) OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) |
OS version
1 2 3 4 5 6 7 |
~ $ cat /etc/os-release NAME="Alpine Linux" ID=alpine VERSION_ID=3.4.4 PRETTY_NAME="Alpine Linux v3.4" HOME_URL="http://alpinelinux.org" BUG_REPORT_URL="http://bugs.alpinelinux.org" |
Get the process list:
1 2 3 4 5 6 |
> top Load average: 2.89 2.89 2.44 3/4217 15717 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 1 0 jhipster S 10762m 16% 3 0% java -Djava.security.egd=file:/dev/./urandom -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=5005,suspend=n -jar /home/jhipster/app.war 15713 15570 jhipster R 1504 0% 13 0% top 15570 0 jhipster S 1512 0% 0 0% sh |
- Or use jps -v commend to list all Java process
- Or use ps aux|grep java
When we try to use top -Hp <pid> to get the CPU usage of all threads in process 6, the error occurred.
1 2 3 4 |
top -Hp 6 top: unrecognized option: H BusyBox v1.24.2 (2016-08-12 14:38:34 GMT) multi-call binary. |
Try to run jcmd <pid> Thread.print under the same user as java process has, otherwise your connections will be dropped. Java doesn’t care if you are root or not.
1 2 |
sudo -u <java_process_user> jcmd <pid> Thread.print sudo -u jhipster jcmd 6 Thread.print |
Or use tool:
A more user-friendly way to view threads per process is via htop, an ncurses-based interactive process viewer. This program allows you to monitor individual threads in tree views.
So we need to instll htop.
1 |
apk add htop |
To enable thread views in htop, launch htop, and press F2 to enter htop setup menu. Choose “Display option” under “Setup” column, and toggle on “Tree view” and “Show custom thread names” options. Presss F10 to exit the setup.
And you can sort the thread list by PERCENT CPU:
When trying with jstack, got the error:
jstack: not found
Now the extra OpenJDK with the toolkit is required:
1 2 3 4 5 6 7 |
# if proxy is required export no_proxy=*.local,localhost,127.0.0.*,10.*,192.168.* export https_proxy=http://user:pwd#proxy.com:8080 export http_proxy=https://user:pwd#proxy.com:8080 apk update apk add openjdk8 |
Now the OpenJDK has been installed
1 2 3 4 5 |
~ $ ls /usr/lib/jvm/ default-jvm java-1.8-openjdk # openjdk location cd /usr/lib/jvm/java-1.8-openjdk |
Try with jstack, still got the error:
1 2 |
jstack -l 1 > /tmp/jvm.stack 1: Unable to open socket file: target process not responding or HotSpot VM not loaded |
Root cause:
because PID 1 is special and it doesn’t handle any signal unless explicitly declared. So the workaround is to have the java process spawned at non PID 1. Here comes Tini into play!
If you keep bash as PID 1, you’d no longer get signals from
docker stopanddocker kill(without having to write traps). With docker 1.13 you can use--initto have docker put in tini as PID 1; it’ll forward signals and reap zombies.
Information:
JVM will generate a directory under /tmp named hsperfdata_$USER($USER is the user name that starts the java process). In this directory, there are PID files, which will store the JVM process information.
Tools like jps、jstack will read the PID files in /tmp/hsperfdata_$USER to get the connection information.
If gets the error：Unable to open socket file. The root cause is that the PID file of the java process has been deleted.
Modify the Dockerfile and rebuild the image.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
FROM openjdk:8-jre-alpine # if proxy is required export no_proxy=*.local,localhost,127.0.0.*,10.*,192.168.* export https_proxy=http://user:pwd#proxy.com:8080 export http_proxy=https://user:pwd#proxy.com:8080 ENV SPRING_OUTPUT_ANSI_ENABLED=ALWAYS \ JHIPSTER_SLEEP=0 \ JAVA_OPTS="" # Add a jhipster user to run our application so that it doesn't need to run as root RUN adduser -D -s /bin/sh jhipster WORKDIR /home/jhipster # Install Tini RUN apk add --no-cache tini RUN apk update RUN apk add openjdk8 RUN apk add sudo RUN apk add htop ENV JAVA8_HOME /usr/lib/jvm/java-1.8-openjdk ENV PATH "$PATH:$JAVA8_HOME/bin" ADD entrypoint.sh entrypoint.sh RUN chmod 755 entrypoint.sh && chown jhipster:jhipster entrypoint.sh USER jhipster ENTRYPOINT ["/sbin/tini", "--", "./entrypoint.sh"] EXPOSE 18082 5701/udp ADD *.jar app.jar |
3. Analysis
Now go into the container and try with htop. Notice that the app process has changed from 1 to 6.
1 2 3 4 5 6 |
Load average: 0.76 1.20 0.92 2/3935 14300 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 6 1 jhipster S 4192m 6% 5 0% java -Djava.security.egd=file:/dev/./urandom -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=5005,suspend=n -jar /home/jhipster/app.jar 14220 0 root S 1516 0% 1 0% sh 14300 14220 root R 1504 0% 7 0% top 1 0 jhipster S 724 0% 1 0% /sbin/tini -- ./entrypoint.sh |
Here I take thread 233 as an example:
3-1. Jstack
Try with jstack, still have a permission issue.
1 2 |
sudo jstack -l 6 > /tmp/jvm.stack 6: well-known file is not secure |
In this case, try this command with the user jhipster, which is used to run this Java application.
1 |
sudo -u jhipster jstack -l 6 > /tmp/jvm.stack |
Copy this stack file from the container:
1 |
docker cp container_id:/tmp/jvm.stack /tmp/jvm.stack |
Remember that we have the top CPU usage thread id 233, now we can convert it to Hexadecimal： 0XE9
Decimal to Hexadecimal: https://www.rapidtables.com/convert/number/hex-dec-bin-converter.html
Then we could find nid=0xe9 in jvm.stack
1 |
cat /tmp/jvm.stack |grep '0xe9' -C 8 |
We can use http://heaphero.io/index.jsp to analyze this stack file
or https://visualvm.github.io/download.html
3-2. jstat
jstat will print the detail of GC: Young GC and Full GC count，stack information:
1 2 3 |
jstat –gcxxx -t pid <interval> <count> jstat -gcutil -t 6 2000 100 |
1 2 3 4 5 6 7 8 |
/home/jhipster # jstat -gcutil -t 6 2000 100 Picked up _JAVA_OPTIONS: -Xmx8192m -Xms2048m Timestamp S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 1432.6 0.00 98.87 22.55 4.66 95.86 94.34 15 0.215 4 0.652 0.867 1434.6 0.00 98.87 22.55 4.66 95.86 94.34 15 0.215 4 0.652 0.867 1436.6 0.00 98.87 22.63 4.66 95.86 94.34 15 0.215 4 0.652 0.867 1438.6 0.00 98.87 22.67 4.66 95.86 94.34 15 0.215 4 0.652 0.867 1440.6 0.00 98.87 22.75 4.66 95.86 94.34 15 0.215 4 0.652 0.867 |
3-3. jmap
jmap prints the Java process heap information.
1 2 3 4 5 |
jmap –heap pid jmap –heap 6 # get error 6: well-known file is not secure |
Dump the heap information to file:
TODO not work!
1 2 3 |
jmap -dump:format=b,file=/tmp/jvm.jmap 6 6: well-known file is not secure |
Info optional:
The –F option can be used when the target process is not responding
1 |
sudo -u jhipster jstack -l -F 233 > /tmp/jvm.stack |
Adding '-XX:+StartAttachListener' to JVM argument fixed the issue.
1 |
-XX:+StartAttachListener |
Reference:
https://blog.csdn.net/shadow_zed/article/details/88167541
https://solidx.dev/java-thread-dump-alpine-docker-container-pid-1/
https://www.cnblogs.com/hujinzhong/p/13301268.html
https://stackoverflow.com/questions/7420501/jstack-target-process-not-responding
https://stackoverflow.com/questions/26140182/running-jmap-getting-unable-to-open-socket-file
![[转]最全面的 Java G1学习笔记](http://www.mobabel.net/wp-content/themes/hueman/assets/front/img/thumb-medium-empty.png)
![[汇总]AI经验](http://www.mobabel.net/wp-content/themes/hueman/assets/front/img/thumb-small-empty.png)
