1. Prerequisite

Java application with docker based on JDK 8 and alpine.

This app is a Jhipster application.

Dockerfile:

entrypoint.sh

 

2. Precheck

Enter container to check the process. Of course, we have only one process from the app.

Enter container with root role.

Java version:

OS version

Get the process list:

1. Or use jps -v commend to list all Java process
2. Or use ps aux|grep java

When we try to use top -Hp <pid> to get the CPU usage of all threads in process 6, the error occurred.

Try to run jcmd <pid> Thread.print under the same user as java process has, otherwise your connections will be dropped. Java doesn’t care if you are root or not.

 

Or use tool:

A more user-friendly way to view threads per process is via htop, an ncurses-based interactive process viewer. This program allows you to monitor individual threads in tree views.

So we need to instll htop.

To enable thread views in htop, launch htop, and press F2 to enter htop setup menu. Choose “Display option” under “Setup” column, and toggle on “Tree view” and “Show custom thread names” options. Presss F10 to exit the setup.

And you can sort the thread list by PERCENT CPU:

 

When trying with jstack, got the error:

jstack: not found

Now the extra OpenJDK with the toolkit is required:

Now the OpenJDK has been installed

Try with jstack, still got the error:

Root cause:

because PID 1 is special and it doesn’t handle any signal unless explicitly declared. So the workaround is to have the java process spawned at non PID 1. Here comes Tini into play!

If you keep bash as PID 1, you’d no longer get signals from docker stop and docker kill (without having to write traps). With docker 1.13 you can use --init to have docker put in tini as PID 1; it’ll forward signals and reap zombies.

Information:

JVM will generate a directory under /tmp named hsperfdata_$USER($USER is the user name that starts the java process). In this directory, there are PID files, which will store the JVM process information.

Tools like jps、jstack will read the PID files in /tmp/hsperfdata_$USER to get the connection information.

If gets the error：Unable to open socket file. The root cause is that the PID file of the java process has been deleted.

 

Modify the Dockerfile and rebuild the image.

3. Analysis

Now go into the container and try with htop. Notice that the app process has changed from 1 to 6.

Here I take thread 233 as an example:

3-1. Jstack

Try with jstack, still have a permission issue.

In this case, try this command with the user jhipster, which is used to run this Java application.

Copy this stack file from the container:

Remember that we have the top CPU usage thread id 233, now we can convert it to Hexadecimal： 0XE9

Decimal to Hexadecimal: https://www.rapidtables.com/convert/number/hex-dec-bin-converter.html

Then we could find nid=0xe9 in jvm.stack

We can use http://heaphero.io/index.jsp to analyze this stack file

or https://visualvm.github.io/download.html

 

3-2. jstat

jstat will print the detail of GC: Young GC and Full GC count，stack information:

 

3-3. jmap

jmap prints the Java process heap information.

Dump the heap information to file:

TODO not work!

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Info optional:

The –F option can be used when the target process is not responding

Adding '-XX:+StartAttachListener' to JVM argument fixed the issue.

 

Reference:

https://blog.csdn.net/shadow_zed/article/details/88167541

https://unixhealthcheck.com/blog?id=465#:~:text=To%20enable%20thread%20views%20in%20htop%2C%20launch%20htop%2C%20and%20press,F10%20to%20exit%20the%20setup.

https://solidx.dev/java-thread-dump-alpine-docker-container-pid-1/

https://www.cnblogs.com/hujinzhong/p/13301268.html

https://stackoverflow.com/questions/7420501/jstack-target-process-not-responding

https://stackoverflow.com/questions/26140182/running-jmap-getting-unable-to-open-socket-file

https://stackoverflow.com/questions/25438983/com-sun-tools-attach-attachnotsupportedexception-unable-to-open-socket-file-ta