Java 11.0.16.0 / 17.0.4.0 suddenly spikes in memory and crashes

Environment

• OpenJDK
  • 11
  • < 17.0.8

Issue

• We recently updated our JDK to 11.0.16.0 or 17.0.4.0. Now at times we see the java process suddenly spikes in memory usage before crashing soon after. hs_err_pid files show large malloc attempts from a jvm compiler thread like below in a compile task for sun.security.ssl.SSLEngineInputRecord::decodeInputRecord

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 1073741840 bytes for Chunk::new
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
#   JVM is running with Zero Based Compressed Oops mode in which the Java heap is
#     placed in the first 32GB address space. The Java Heap base address is the
#     maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
#     to set the Java Heap base and to place the Java Heap above 32GB virtual address.
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (arena.cpp:197), pid=2907, tid=2927
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-11.0.16.0.8-1.el7_9) (11.0.16+8) (build 11.0.16+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-11.0.16.0.8-1.el7_9) (11.0.16+8-LTS, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x00007f5134b1b000):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=2927, stack(0x00007f5138229000,0x00007f513832a000)]


Current CompileTask:
C2:299829840 17165   !   4       sun.security.ssl.SSLEngineInputRecord::decodeInputRecord (812 bytes)

Stack: [0x00007f5138229000,0x00007f513832a000],  sp=0x00007f5138324360,  free space=1004k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xf077e2]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1c2
V  [libjvm.so+0xf08793]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x43
V  [libjvm.so+0x6fc210]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0x110
V  [libjvm.so+0x47d741]  Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x1f1
V  [libjvm.so+0x47d982]  Arena::Arealloc(void*, unsigned long, unsigned long, AllocFailStrategy::AllocFailEnum)+0x1f2
V  [libjvm.so+0xc2417a]  Node::clone() const+0x29a
V  [libjvm.so+0xde42d0]  PhaseIdealLoop::split_up(Node*, Node*, Node*) [clone .part.114]+0xc50
V  [libjvm.so+0xde6090]  PhaseIdealLoop::do_split_if(Node*)+0x660
V  [libjvm.so+0xafb9f9]  PhaseIdealLoop::split_if_with_blocks(VectorSet&, Node_Stack&, bool)+0x129
V  [libjvm.so+0xaf61a5]  PhaseIdealLoop::build_and_optimize()+0x11e5
V  [libjvm.so+0x69b05b]  Compile::optimize_loops(int&, PhaseIterGVN&, LoopOptsMode)+0x2bb
V  [libjvm.so+0x69cb3b]  Compile::Optimize()+0xb5b
V  [libjvm.so+0x69e61d]  Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, bool, DirectiveSet*)+0x11ad
V  [libjvm.so+0x5bffb4]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x124
V  [libjvm.so+0x6a765b]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x43b
V  [libjvm.so+0x6a8e68]  CompileBroker::compiler_thread_loop()+0x598
V  [libjvm.so+0xe9ee72]  JavaThread::thread_main_inner()+0x232
V  [libjvm.so+0xe9b1ca]  Thread::call_run()+0x16a
V  [libjvm.so+0xc5d8d8]  thread_native_entry(Thread*)+0xe8

Resolution

• Upgrade to OpenJDK 17.0.8+ / 21 or 11.0.18+
• Add the following JVM option on impacted versions as a workaround:

-XX:CompileCommand=exclude,sun/security/ssl/SSLEngineInputRecord,decodeInputRecord

• Downgrade the JDK to the previously used version as a workaround

Root Cause

Content from bugs.openjdk.org is not included.JDK-8292301

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.