JVM crash in libc.so with large heap size > 128 G when using the CMS collector
Environment
- Red Hat Enterprise Linux (RHEL) 5, 5.6, 5.9
- Java
- Oracle JDK 1.6.0 update 45
- OpenJDK JRE version: 6.0_17-b17 (IcedTea6 1.7.5)
- Concurrent Mark Sweep (CMS) Collector
Issue
- Using OpenJDK1.6 with concurrent GC, heap size specified works with 128GB, but would abort if you specify a 300GB heap size. JVM crashes at memset+0x40 with the following exception
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000003fcc67ae40, pid=8452, tid=1080600896
#
# JRE version: 6.0_17-b17
# Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
# Derivative: IcedTea6 1.7.5
# Distribution: Custom build (Tue Oct 12 00:18:31 EDT 2010)
# Problematic frame:
# C [libc.so.6+0x7ae40] memset+0x40
.. ..
VM Arguments: jvm_args: -Xms307200m -Xmx307200m -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31
- We are getting a segmentation violation when we try to run a java process with a heap size of more than 128 GB on RHEL 5.9. Works with 128 heap size GB but fails with below error with 129 GB heap or larger. System has 256 GB of physical RAM.
# JRE version: 6.0_45-b06
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.45-b01 mixed mode linux-amd64 )
# Problematic frame:
# C [libc.so.6+0x7a050] unsigned long+0x40
#
# An error report file with more information is saved as:
# /app/log/error/hs_err_pid9240.log
.. .. ..
Java Options:
JavaOpts=" d64 -Xms130g -Xmx130g
-XX:CompileThreshold=8000 -XX:NewSize=384m -XX:MaxNewSize=384m -XX:PermSize=256m -XX:MaxPermSize=256m
-XX:SurvivorRatio=128 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=94
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:$ApplHome/logs/crt-gc-`date '+%Y%m%d%H%M%S'`.log -XX:-TraceClassUnloading
- JVM crashes with for large heap size when using the Concurrent Mark Sweep (CMS) collector
Resolution
- Red Hat is working on a solution for OpenJDK 1.6
- Try the latest release of JDK 1.7
Root Cause
Starting the JVM with -Xms150G -Xmx150G or greater will cause it to segfault.
Diagnostic Steps
Get a corefile when the crash occurs.
Core was generated by `/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/bin/java -Xms307200m -Xmx3072'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003fcd20aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) t 19
[Switching to thread 19 (Thread 0x40100940 (LWP 19313))]#0 0x0000003fcc67ae40 in memset () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003fcc67ae40 in memset () from /lib64/libc.so.6
#1 0x00002aaaaaceb3e4 in set_offset_array (this=0x63c5c8, start_card=<value optimized out>, end_card=628378879)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/blockOffsetTable.hpp:150
#2 BlockOffsetArray::set_remainder_to_point_to_start_incl (this=0x63c5c8, start_card=<value optimized out>, end_card=628378879)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/blockOffsetTable.cpp:198
#3 0x00002aaaaaceb753 in BlockOffsetArray::BlockOffsetArray (this=0x63c5c8, array=<value optimized out>, mr=..., init_to_zero=160)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/blockOffsetTable.cpp:114
#4 0x00002aaaaad520dd in BlockOffsetArrayNonContigSpace (this=0x63c550, bs=0x47, mr=..., use_adaptive_freelists=true,
dictionaryChoice=dictionaryBinaryTree) at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/blockOffsetTable.hpp:363
#5 CompactibleFreeListSpace::CompactibleFreeListSpace (this=0x63c550, bs=0x47, mr=..., use_adaptive_freelists=true,
dictionaryChoice=dictionaryBinaryTree)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/gc_implementation/concurrentMarkSweep/compactibleFreeListSpace.cpp:54
#6 0x00002aaaaad83ca1 in ConcurrentMarkSweepGeneration::ConcurrentMarkSweepGeneration (this=0x63c360, rs=...,
initial_byte_size=<value optimized out>, level=<value optimized out>, ct=<value optimized out>, use_adaptive_freelists=true,
dictionaryChoice=dictionaryBinaryTree)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/gc_implementation/concurrentMarkSweep/concurrentMarkSweepGeneration.cpp:178
#7 0x00002aaaaae29c8f in GenerationSpec::init (this=<value optimized out>, rs=..., level=1, remset=<value optimized out>)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/generationSpec.cpp:60
#8 0x00002aaaaae1e16f in GenCollectedHeap::initialize (this=0x618820)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/genCollectedHeap.cpp:137
#9 0x00002aaaab14e7eb in Universe::initialize_heap () at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/universe.cpp:772
#10 0x00002aaaab14fe30 in universe_init () at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/memory/universe.cpp:701
#11 0x00002aaaaae524d5 in init_globals () at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/runtime/init.cpp:91
#12 0x00002aaaab13da5b in Threads::create_vm (args=0x401000d0, canTryAgain=0x40100057)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/runtime/thread.cpp:2900
#13 0x00002aaaaae99464 in JNI_CreateJavaVM (vm=0x401000f8, penv=0x401000f0, args=0x4747474747474747)
at /usr/src/debug/icedtea6-1.7.5/openjdk/hotspot/src/share/vm/prims/jni.cpp:3263
#14 0x00000000004026c6 in InitializeJVM (_args=<value optimized out>) at ../../../../src/share/bin/java.c:1296
#15 JavaMain (_args=<value optimized out>) at ../../../../src/share/bin/java.c:431
#16 0x0000003fcd20673d in start_thread () from /lib64/libpthread.so.0
#17 0x0000003fcc6d40cd in clone () from /lib64/libc.so.6
Product(s)
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.