Possible causes of high CPU usage in Java apps might be related to:
Garbage Collector (GC) executing Major or Full collections too frequently without freeing much memory as a result of a memory leak in one of the applications served by the servlet container or a leak in the servlet container itself.
GC often executing Major or Full collections (similar to previous suggestion) because in fact, the application needs more memory.
Issues with the application like resource contention, long running jobs, expensive computation, …
The intention of this entry is to document a simple process to troubleshoot high CPU usage in Java apps once it has been observed that it’s not related to GC.
2. FIND THE JVM PROCESS
3. GENERATE THREAD DUMP OF THE JAVA MAIN THREAD FOUND IN 2.
JVM main thread = Linux process, other Java threads = Lightweight process (LWP)
4. FIND SUSPECT LIGHTWEIGHT PROCESSES (LWPs) WITH HIGH CPU USAGE
It can be observed there are two LWPs using 32% and 14% of the CPU where the rest stays around the same percentage.
5. CONVERT LWP IDs FROM DECIMAL TO HEXADECIMAL FOR PIDs WITH HIGH CPU USAGE
6. OPEN THREADDUMP FROM 3. AND LOCATE HEX VALUES 0x67ed and 0x67ea
From previous stack trace samples, seems high CPU usage is related to log4j async appender, in this case very likely involving PCIPatternLayout.format() method.
In both instances I was involved with such troubleshooting, the solution was to uplift snmp4j to the latest version at the time of the issue was found and to remove custom layout from log4j appender configuration file for this application.
If you found this post helpful and would like to receive updates when content like this gets published, sign up to the newsletter.