Java java.lang.OutOfMemoryError "Out of swap space?"

Solution Verified - Updated

Environment

  • Java

  • 32-bit

Issue

  • After running an application for about a week, the following error appears in the JBoss EAP server log: 
    Exception java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
  • JBoss crashes with the JVM creating a fatal error log with the following header and current thread information:
    #
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 270480 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (allocation.cpp:218), pid=6932, tid=8336
#  Error: Chunk::new
#
# Java VM: Java HotSpot(TM) Server VM (11.0-b15 mixed mode windows-x86)
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

    ---------------  T H R E A D  ---------------
    

    Current thread (0x64075800):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=8336, stack(0x643d0000,0x64420000)]

  • The fatal error log suspiciously shows plenty of available physical memory and swap space:

    Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)

Resolution

Reduce memory demands or increase available address space.

Reduce Memory
  • Explicitly set or reduce the thread stack size (e.g. to 128k). See How do I set Java thread stack size. If you get a StackOverflowError, then increase by increments of 64k until the StackOverflowError goes away.
  • Based on the output from the GC logs, determine whether the permgen (-XX:MaxPermSize) and heap (-Xmx) can be lowered.
  • If using JDK 1.5, upgrade to JDK 1.6 as it has a lower memory foot (due to more efficient memory mapping of jar files that significantly reduces address space needs. JDK 15 memory maps entire jar files, whereas Content from weblogs.java.net is not included.JDK and OpenJDK 1.6 only memory maps a central directory
  • If the issue occurs in a compiler thread, exclude the problem method from compilation, for example:
-XX:CompileCommand=exclude,com/my/package/ClassName,methodName
Increase Address Space

Root Cause

  • Native heap address space exhaustion.
  • The theoretical limit for a 32-bit process is 4GB. However, the available address space is likely to be much less than 4GB. Process address space is split into user space and kernel space, with most OSes mapping some amount of process space to a common kernal memory area. In addition address space must be contiguous. Shared libraries such as Windows DLLs can have preferred memory loading locations, which fragments the address space and further reduces the maximum address space that can be allocated.
  • Fragmented physical memory. There is not a large enough contiguous block to allocate the requested amount.
  • Compiler bug resulting in too much memory allocation.

Diagnostic Steps

General

  • When does the issue happen? Is there any pattern (e.g. on startup, after running a certain amount of time, when a particular use case is executed, etc.)?

  • Can the issue be reproduced on demand?

  • Does the application use native code?

  • Is the application installed in a dedicated environment, or are there other applications and/or services that could potentially have shared libraries fragmenting address space?

  • What JVM options are being used? Is any instrumentation (e.g. -javaagent) being used?

  • Does the application generate a lot of threads? Get a thread dump when the issue happens:

  • Is NIO being used (e.g. JBossWeb or Tomcat NIO connectors)?

  • Is the Java application running in a physical or virtual (e.g.) environment?

  • How much physical memory (RAM)?

  • What is the maximum address space size?

  • If a fatal error log is not being created, enable garbage collection logging. How much heap + perm gen space was used at the time the issue happened?

  • What is the Java process size (heap + perm gen + stack + native) at the point of failure?

  • How does the Java process size compare to the amount of heap + perm gen space used?

  • How does the the Java process size compare to the amount of physical memory indicated in the fatal error log?

OpenJDK / Sun JDK
  • Check to see if a fatal error log was created. The fatal error log  will be in the directory defined by the user.dir environment variable in  boot.log (typically $JBOSS_HOME/bin/) and named  hs_err_pid<pid>.log (where <pid> is the JBoss process ID).  If you expect one but it isn't there, check that the user has write  access to that directory.
  • Content from java.sun.com is not included.Review the fatal error log to determine where the issue is originating from (e.g. native code, Java bytecode, etc.), JVM version, OS, hardware  (e.g. memory, cpu), JVM options, heap usage, etc. Note any common factors amongst crashes (e.g. the same method being compiled by the JIT compiler, similar heap usage, etc.).
  • Check if the fatal error log shows the crash when the JIT compiler tries to compile the following Saxon method. This is an issue when using Saxon 9.1.x.
    Current thread (0x00000000067f6000):  JavaThread "CompilerThread0" daemon [_thread_in_native, id=2200, stack(0x0000000006990000,0x0000000006a90000)]

    Stack: [0x0000000006990000,0x0000000006a90000]
    

    Current CompileTask:
    
C2:3670  !   net.sf.saxon.event.ReceivingContentHandler.startElement(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Lorg/xml/sax/Attributes;)V (638 bytes)

###### JBoss
  • Obtain boot.log and server.log.
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.