Poor performance and sporadic high CPU usage of Java 7 application due to compiler cache exhaustion

Environment

• OpenJDK 1.7.0
• Oracle JDK 1.7.0

Issue

• A terribly slow web application on EAP
• Sporadic high CPU usage
• Severe performance degradation after some hours
• Performance bottleneck on a common method like java.lang.String.equalsIgnoreCase()

Resolution

Increase the code cache size by ReservedCodeCacheSize JVM option. The default size is 48 MB in Java 7 while it is 240 MB in Java 8.

JAVA_OPTS="$JAVA_OPTS ... -XX:ReservedCodeCacheSize=240m ..."

The optimal size depends on the application running. Set it higher enough than the maximum usage of the code cache. See "Diagnostic Steps" for how to monitor it. 240 MB should be safe for most cases.

Root Cause

HotSpot JVM of Java 7 has a relatively small code cache size by default. The code cache size, which is observable by JConsole, stores JIT compiled code and the size can be small for a large Java application like EAP with web applications deployed. Once the cache hits the limit, a cache flushing algorithm runs to use the cache efficiently. But there is a bug in the flushing algorithm¹ and it causes even a frequently used method (i.e., hot spot) be removed from the cache. The method is decompiled and runs in interpreter mode². It makes the Java program slow and CPU bound and causes sporadic high CPU usage.

Diagnostic Steps

Connect to the java process by JConsole. Monitor "Memory Pool Code Cache" in Memory tab. Note that the current usage is not important. You should check the historical high.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.