JVM crash failing native memory allocation(malloc)

Solution Verified - Updated

Environment

  • OpenJDK
    • Red Hat build of OpenJDK

Issue

After increasing JVM heapsize from 1G to 2G, the JVM was crashed with following hs_err log.

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.cpp:211), pid=2536, tid=38
#
# JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode solaris-amd64 compressed oops)
# Core dump written. Default location: $EAP_HOME/domain/core or core.2536
#
...

OS: Oracle Solaris 10 1/13 s10x_u11wos_24a X86
uname:SunOS 5.10 Generic_Virtual i86pc
Memory: 4k page, physical 268299192k(246435252k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (24.80-b11) for solaris-amd64 JRE (1.7.0_80-b15), built on Apr 10 2015 11:47:02 by "" with Sun Studio 12u1
...
  • Native memory allocation (mmap) failed to map in the fatal error log:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 262144 bytes for committing reserved memory
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap.
...
#  Out of Memory Error (os_solaris.cpp:2563), pid=8544, tid=0x0000000000004813
#
# JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode solaris-sparc compressed oops)
  • Out of swap space to map in thread stack in the fatal error log:
# There is insufficient memory for the Java Runtime Environment to continue.
# Out of swap space to map in thread stack.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
...
#
#  Out of Memory Error (os_solaris_sparc.cpp:570), pid=1129, tid=0x0000000000008488
#
# JRE version: Java(TM) SE Runtime Environment (8.0_231-b11) (build 1.8.0_231-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.231-b11 mixed mode solaris-sparc compressed oops)

Resolution

Workarounds for this situation:

  • Run with -XX:-UseCompressedOops (keeping in mind that you would lose the performance benefit that comes with enabling CompressedOops). This will instruct the JVM to run without the Compressed Oops feature and it would not try to fit the Java Heap in the first 4GB of heap.

or

  • Keep the CompressedOops and set the base of the Java Heap with the JVM option -XX:HeapBaseMinAddress=n to specify the address where the Java Heap should start from. If you are running out of native memory, set it to a higher address so that there is enough room available for the native mallocs, and if Java Heap is getting exhausted then set it to a lower value. And if you need java heap to be allocated beyond 4GB or 32GB then HeapBaseMinAddress should be set to a value so that HeapBaseMinAddress + heap size > 4GB or 32GB.
    The default value of HeapBaseMinAddress on most of the platforms is 2GB.

Root Cause

Diagnostic Steps

Check the heap address in the fatal error log.
For example:

heap address: 0x00000006c2a00000, size: 4062 MB, Compressed Oops mode: Zero based, Oop shift amount: 3

0x00000006c2a00000 hex = 29035069440 decimal = 27g. This is the limit of the native heap.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.