java.lang.OutOfMemoryError: Metaspace

Solution Verified - Updated

Environment

  • OpenJDK
    • Red Hat Build of OpenJDK
    • Oracle JDK

Issue

  • java.lang.OutOfMemoryError: Metaspace
  • Out of Memory Metaspace errors shortly after startup or certain CLI commands;

Resolution

Either remove or increase -XX:MaxMetaspaceSize:

  1. Remove -XX:MaxMetaspaceSize and let the JVM size the metaspace automatically.

Unlike perm gen, it is typically not necessary to set a max/min Metaspace size for the following reasons:

  • Metaspace size is unlimited by default. Since it only holds class metadata, size requirements are minimal.
  • It does not require a full gc to resize the metaspace, as it did with the perm generation.

However, if the application is experiencing large metaspace leaks, it may be preferable for the application to fail rather than keep increasing memory size.

  1. Increase the Metaspace size using the -XX:MaxMetaspaceSize JVM argument. For example:
-XX:MaxMetaspaceSize=2g

When UseCompressedOops and UseCompressedClassesPointers are enabled (default), MaxMetaspaceSize includes the CompressedClassSpaceSize.

For example, the following options will result in a 2gb space for class metadata, with 1gb for compressed class pointers:

-XX:MetaspaceSize=2g -XX:MaxMetaspaceSize=2g -XX:CompressedClassSpaceSize=1g

If MaxMetaspaceSize is set smaller than CompressedClassSpaceSize, the JVM auto adjusts CompressedClassSpaceSize as follows:

CompressedClassSpaceSize = MaxMetaspaceSize - 2 * InitialBootClassLoaderMetaspaceSize

For example, the following options will result in a 512M space for class metadata, with 504M (assuming 4M default InitialBootClassLoaderMetaspaceSize) for compressed class pointers:

-XX:MetaspaceSize=512M -XX:MaxMetaspaceSize=512M -XX:CompressedClassSpaceSize=1024M

See:

  1. Set jsp-config development "false" (or remove development="true", as "false" is the default value):
<jsp-config development="false" tag-pooling="false" check-interval="1" modification-test-interval="1" recompile-on-fail="true"/>

Root Cause

This behavior can happen due to several issues, see the most common bellow:

Diagnostic Steps

Java Application Log Analysis

Verify if the OutOfMemoryError (OOME) is due to Metaspace.

GC Log Analysis

Verify a full collection does not reclaim Metaspace.

Classloader Stats Analysis

JDK8 u13+

  • Enable -XX:+UnlockDiagnosticVMOptions then capture GC.class_stats with jcmd (JDK 1.8 u13+):

    $JAVA_HOME/bin/jcmd <JBOSS_JAVA_PID> GC.class_stats -all -csv > class_stats.csv

    Notes:

    • Forces a full gc.
    • Open class_stats.csv in a spreadsheet (e.g. LibreOffice Calc).
    • Check InstBytes column for largest retainers.
    • Review the largest retainers in the heap dump to look for clues. See below.

JDK 1.8 u13-

  • Get clstats output from a time of high Metaspace piped to a file: 

    $JAVA_HOME/bin/jmap -clstats <JAVA_PID> > clstats.out

    Example output:

    Attaching to process ID 12345, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.101-b13
class_loader    classes bytes   parent_loader   alive?  type

<bootstrap>     2631    4705272   null          live    <internal>
0x00000000c91be518      1       1455    0x00000000fdcf0978      dead    sun/reflect/DelegatingClassLoader@0x00000001000099d8
0x00000000c377a770      0       0       0x00000000fd618000      live    org/eclipse/osgi/internal/loader/EquinoxClassLoader@0x00000001000bf438
0x00000000c64f4a48      454     772114  0x00000000fd618000      live    org/eclipse/osgi/internal/loader/EquinoxClassLoader@0x00000001000bf438
...
0x00000000c7651c90      1       864     0x00000000c35c1d98      dead    sun/reflect/DelegatingClassLoader@0x00000001000099d8
0x00000000c2ff5108      54      136855  0x00000000fd618000      live    org/eclipse/osgi/internal/loader/EquinoxClassLoader@0x00000001000bf438
0x00000000c76f9898      1       1455    0x00000000c3eb0f60      dead    sun/reflect/DelegatingClassLoader@0x00000001000099d8

total = 426     31560   64301535            N/A         alive=277, dead=149         N/A

    Notes:

    • Do not use this for JDK 1.8 u13+. It does not seem to provide useful information.
    • Requires debuginfo.
    • Requires running jmap as the user that started the Java process.
    • Will put the JVM in a safepoint (all threads stopped) while running.
    • Make take several minutes to produce output.
    • Forces a full gc.

Heap Dump Analysis

Get a heap dump when OOME happens using the -XX:+HeapDumpOnOutOfMemoryError option and check the following:

  • Use the following Object Query Language (OQL) query to find duplicate deployments (e.g. duplicate application war/ear/jars):

        #EAP6
    SELECT module.identifier.name.value.toString() FROM org.jboss.modules.ModuleClassLoader

    #EAP7
    SELECT module.name.value.toString() FROM org.jboss.modules.ModuleClassLoader

  • Check for multiple instances of org.apache.cxf.common.jaxb.JAXBContextCache$2.

  • Review the classes that classloader stats analysis showed are the largest retainers for clues. For example, what are the immediate dominators? What are the GC root(s)?

  • Check the stack traces of running threads and review for patterns.

Metaspace OOME Report

JDK8 debug builds with -verbose:gc -XX:+TraceMetadataChunkAllocation and JDK11+ builds with standard recommended gc logging options will result in the following report output at the end of the gc logging that can be used for troubleshooting:

Metaspace (data) allocation failed for size 11

 Usage:
   Non-class:    218.74 MB capacity,   209.07 MB ( 96%) used,     9.05 MB (  4%) free+waste,   636.56 KB ( <1%) overhead.
       Class:     33.97 MB capacity,    28.95 MB ( 85%) used,     4.71 MB ( 14%) free+waste,   310.88 KB ( <1%) overhead.
        Both:    252.71 MB capacity,   238.02 MB ( 94%) used,    13.76 MB (  5%) free+waste,   947.44 KB ( <1%) overhead.

 Virtual space:
   Non-class space:      222.00 MB reserved,     221.38 MB (>99%) committed
       Class space:      248.00 MB reserved,      34.62 MB ( 14%) committed
              Both:      470.00 MB reserved,     256.00 MB ( 54%) committed

 Chunk freelists:
    Non-Class:  2.62 MB
        Class:  672.00 KB
         Both:  3.28 MB

 MaxMetaspaceSize: 256.00 MB
 CompressedClassSpaceSize: 248.00 MB
 Initial GC threshold: 96.00 MB
 Current GC threshold: 256.00 MB
 CDS: on

Check if jsp-config development is set to "true":

<jsp-config development="true" ... />

jcmd VM.metaspace available in JDK11+

Starting with JDK11, even without special java startup options, one can use jcmd <pid> VM.metaspace show-loaders=true show-classes=true command to check the number of loaded classes and Metaspace consumption for each class loader. This allows you to identify the primary factors driving Metaspace usage, such as duplicate or dynamic class loading:

 443: CLD 0x0000563e2c6b1cd0: "deployment.app.war" instance of org.jboss.modules.ModuleClassLoader
      Loaded classes:
         1:    com.redhat.gss.example.servlet.ServletA
         2:    com.redhat.gss.example.servlet.ServletB
         3:    com.redhat.gss.example.servlet.ServletC
         ...
        -total-: 35 classes
  Non-Class:    5 chunks,     80.00 KB capacity,    64.09 KB ( 80%) used,    15.55 KB ( 19%) free,    40 bytes ( <1%) waste,   320 bytes ( <1%) overhead, deallocated: 1 blocks with 24 bytes
      Class:    5 chunks,     40.00 KB capacity,    28.84 KB ( 72%) used,    10.84 KB ( 27%) free,     0 bytes (  0%) waste,   320 bytes ( <1%) overhead, deallocated: 4 blocks with 1.73 KB
       Both:   10 chunks,    120.00 KB capacity,    92.94 KB ( 77%) used,    26.40 KB ( 22%) free,    40 bytes ( <1%) waste,   640 bytes ( <1%) overhead, deallocated: 5 blocks with 1.75 KB
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.