JVM crash due to corrupted ParBitMap on RHEL OpenJDK

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux 6.2
  • java-1.6.0-openjdk-1.6.0.0-1.41.1.10.4.el6.x86_64

Issue

  • The fatal error log shows the JVM crashes with the current thread GCTaskThread:
Current thread (0x00007fd1b0014000):  GCTaskThread
  • JVM crashed by SIGSEGV(SEGV_ACCERR) and core dumped as follows.
(gdb) bt
 #0  0x0000003aaf232885 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
 #1  0x0000003aaf234065 in abort () at abort.c:92
 #2  0x00007fd1b86602e9 in os::abort (dump_core=true) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:1558
 #3  0x00007fd1b87942eb in VMError::report_and_die (this=0x7fd1b419b4e0) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/utilities/vmError.cpp:962
 #4  0x00007fd1b8664283 in JVM_handle_linux_signal (sig=11, info=0x7fd1b419b6b0, ucVoid=0x7fd1b419b580, abort_if_unrecognized=-1273383840)
     at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/os_cpu/linux_x86/vm/os_linux_x86.cpp:528
 #5  <signal handler called>
 #6  par_set_bit (this=0x7fd1b8bbfda0, addr=<value optimized out>, size=<value optimized out>) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/utilities/bitMap.inline.hpp:57
 #7  ParMarkBitMap::mark_obj (this=0x7fd1b8bbfda0, addr=<value optimized out>, size=<value optimized out>)
     at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/parMarkBitMap.cpp:95
 #8  0x00007fd1b847ac22 in mark_obj (this=<value optimized out>, cm=0x7fd1b0073d90) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/parMarkBitMap.hpp:375
 #9  mark_obj (this=<value optimized out>, cm=0x7fd1b0073d90) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.hpp:1263
 #10 mark_and_push<narrowOop> (this=<value optimized out>, cm=0x7fd1b0073d90) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.hpp:1312
 #11 instanceKlass::follow_static_fields (this=<value optimized out>, cm=0x7fd1b0073d90) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/oops/instanceKlass.cpp:1716
 #12 0x00007fd1b8487ed7 in instanceKlassKlass::oop_follow_contents (this=0x700000250, cm=0x7fd1b0073d90, obj=0x702243048)
     at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/oops/instanceKlassKlass.cpp:133
 #13 0x00007fd1b86a03da in follow_contents (this=0x7fd1b0073d90) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/oops/oop.pcgc.inline.hpp:98
 #14 ParCompactionManager::follow_marking_stacks (this=0x7fd1b0073d90) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psCompactionManager.cpp:146
 #15 0x00007fd1b868e818 in MarkFromRootsTask::do_it (this=<value optimized out>, manager=<value optimized out>, which=<value optimized out>)
     at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/pcTasks.cpp:133
 #16 0x00007fd1b8432a4f in GCTaskThread::run (this=0x7fd1b0014000) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/gcTaskThread.cpp:136
 #17 0x00007fd1b86618d2 in java_start (thread=0x7fd1b0014000) at /usr/src/debug/icedtea6-1.10.4/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:856
 #18 0x0000003aaf6077f1 in start_thread (arg=0x7fd1b419c700) at pthread_create.c:301
 #19 0x0000003aaf2e570d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
  • The crash happens in par_set_bit() trying to access invalid memory.

Resolution

Disable the use of large pages with the -XX:-UseLargePages option.

It should also be noted that Red Hat support for OpenJDK 6 (1.6) ended in December 2016, for details please refer to OpenJDK Life Cycle and Support Policy. Upgrading to a newer OpenJDK version could be considered, but needs verification/testing if the Java code is compatible with the newer OpenJDK version.

Root Cause

The ParCompactionManager::_mark_bitmap data structure got corrupted. The JVM crashed while processing sections of the heap in the GC mark phase, and tried to access invalid memory in the marking bitmap. Please refer to Content from blogs.oracle.com is not included.Content from blogs.oracle.com is not included.https://blogs.oracle.com/poonam/entry/uselargepages_on_linux for more details.

Diagnostic Steps

See Java Crash in GCTaskThread.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.