Solving OpenJDK C2 compilation issues
Environment
- OpenJDK 8
- OpenJDK 11
- OpenJDK 17
Issue
- How to solve C2 compilation issues?
- Crash on C2 on method get from a certain class?
Resolution
Disclaimer: Links contained herein to an external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.
If the case is c2 compilation blocked thread issue, the first suggestion would be to test with the latest version of OpenJDK and OracleJDK.
The second is to test with c2 compilation disabled completely using:
-XX:TieredStopAtLevel=1
If the problem is Full CodeCache, as shown on the logs (for OracleJDK):
Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= XXX
Which can be diagnosed using JConsole, on the Memory tab, on Memory Pool code cache, which is designed to stores the JVM's bytecode compiled into native code.
The solution to Full CodeCache is to increase the initial cache size and enable flush:
-XX:ReservedCodeCacheSize=512m
-XX:+UseCodeCacheFlushing
Finally, in case none of those steps solved the issue, use the diagnostics steps to verify in depth the issue, find which function is blocking and exclude it from C2 compiler optimization using the command:
-XX:CompileCommand
Root Cause
In the case described here, the method exampleMethod is stuck and this shows on the top -H, as well as the thread got stuck, which is shown in jstack.
Diagnostic Steps
C2 compiler optimization is briefly described on the Content from openjdk.java.net is not included.HotSpot Glossary.
To analyze an issue with C2 compilation, first, use top -H to see details on the high consuming threads.
# top -H
PID USER PR VIRT RES SHR S %CPU %MEM TIME+ COMMAND
68 root 20 441124 345740 10512 R 88.7 1.1 2328:34 java
71 root 20 441124 345740 10512 R 85.0 1.1 2708:41 java
66 root 20 441124 345740 10512 R 7.6 1.1 38:21.01 java
At this point, it is possible to see the thread status using jstack PID command:
"C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0xeb0f5c00 nid=0x44 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE"C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0xeb0fa800 nid=0x47 runnable [0x00000000]
java.lang.Thread.State: RUNNABLE
Next, then go down the stack using the information use jstack -m PID (or Content from linux.die.net is not included.pstack) to see the details on the thread nid (Native Thread ID on Linux it's the PID of the thread)
Thread 1 (process 68):
#0 0xf6abda90 in MemBarCPUOrderNode::Opcode() const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#1 0xf6e8462c in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#2 0xf6e84afb in LoadNode::Value(PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#3 0xf6f5368a in PhaseIterGVN::transform_old(Node*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#4 0xf6f51084 in PhaseIterGVN::optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#5 0xf6aece24 in Compile::Optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#6 0xf6aeea0a in Compile::Compile(ciEnv*, C2Compiler*, exampleMethod*, int, bool, bool, bool) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#7 0xf6a43a4d in C2Compiler::compile_method(ciEnv*, exampleMethod*, int) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#8 0xf6af92b1 in CompileBroker::invoke_compiler_on_method(CompileTask*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#9 0xf6afa0c7 in CompileBroker::compiler_thread_loop() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#10 0xf7031f48 in compiler_thread_entry(JavaThread*, Thread*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#11 0xf703cadf in JavaThread::thread_main_inner() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#12 0xf703cc5b in JavaThread::run() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#13 0xf6f0c179 in java_start(Thread*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so
#14 0xf77b8b2c in start_thread () from /lib/libpthread.so.0
#15 0xf76cf17e in clone () from /lib/libc.so.6
On the line #7, the line shows C2Compiler::compile_method, which is the method that is being compiled, on our case is exampleMethod.
At this point, by analyzing the stack trace, it is possible to disable to see if the problem is in one function particularly. If that's the case, one can use -XX:CompileCommand flag to exclude a certain method for being optimized by C2 compiler, e.g exampleMethod.
Then, it is necessary to investigate the function: exampleMethod to see more details.
If that's not information enough, the next step is to get the core dump (and binaries) with gdb and see the core dump, the follow solution can help core dump analysis.
Printing Compilation details:
Use the flag PrintCompilation to display compilation details when running a process, example:
# using c2 compilation
$ java -XX:+PrintCompilation Example
# only c1 compilation
$ java -XX:+PrintCompilation -XX:TieredStopAtLevel=1 Example
Example output:
56 36 % 4 Example::iterateWithDynamicSize @ 9 (57 bytes)
59 33 % 3 Example::iterateWithDynamicSize @ -2 (57 bytes) made not entrant
59 31 % 3 Example::iterateWithConstantSize @ -2 (59 bytes) made not entrant
59 37 4 Example::iterateWithDynamicSize (57 bytes)
59 38 4 Example::iterateWithConstantSize (59 bytes)
Interpreting PrintCompilation Output
| Column | Meaning |
|---|---|
| 56 | Timestamp |
| 36 (%) | compilation_id and method_attributes (% for OSR compilations) |
| 4 | Max level of compilation (if -XX:TieredStopAtLevel=1, so then all will be 1) |
| Example::iterateWithDynamicSize | Method name |
| @ 9 | For OSR compilations osr_bci |
| (57 bytes) | Size of the compilation unit in bytecode (not the size of the compiled code) |
If one removes Tiered Compilation, with using -XX:-TieredCompilation, so then the max level will be empty:
java -XX:+PrintCompilation -XX:-TieredCompilation Test
45 1 java.lang.String::hashCode (55 bytes)
55 2 n java.lang.System::currentTimeMillis (native) (static)
55 3 Test::call (17 bytes)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.