Java heap dump not created on OutOfMemoryError

Environment

• OpenJDK
• Oracle/Sun JDK

Issue

JBoss EAP contains -XX:+HeapDumpOnOutOfMemoryError in the JVM startup options; however, when an OutOfMemoryError occurs, no heap dump appears to be created. The only message captured in the console is:

JBossAS process (<JAVA_PID>) received KILL signal

Resolution

• The application, not the JVM, creates and throws the OutOfMemoryError.

• The heap dump is written only on the first OutOfMemoryError. If the application chooses to catch it and continues to run, the next OutOfMemoryError will not cause a heap dump to be written.

• The OutOfMemoryError is of type "java.lang.OutOfMemoryError: GC overhead limit exceeded" or "java.lang.OutOfMemoryError: unable to create new native thread", neither of which initiate a heap dump.

• If the OnOutOfMemoryError parameter is also set, the heap dump should be taken before OnOutOfMemoryError is executed.

  According to the source code of OpenJDK, generating dump by HeapDumpOnOutOfMemoryError should get executed before executing whatever is passed to OnOutOfMemoryError:

  • hotspot/src/share/vm/utilities/debug.cpp

        void report_java_out_of_memory(const char* message) {
          static jint out_of_memory_reported = 0;

          // A number of threads may attempt to report OutOfMemoryError at around the
          // same time. To avoid dumping the heap or executing the data collection
          // commands multiple times we just do it once when the first threads reports
          // the error.
          if (Atomic::cmpxchg(1, &out_of_memory_reported, 0) == 0) {
            // create heap dump before OnOutOfMemoryError commands are executed
            if (HeapDumpOnOutOfMemoryError) {
              tty->print_cr("java.lang.OutOfMemoryError: %s", message);
              HeapDumper::dump_heap_from_oome();
            }

            if (OnOutOfMemoryError && OnOutOfMemoryError[0]) {
              VMError err(message);
              err.report_java_out_of_memory();
            }
          }
        }

Root Cause

• Another resource like threads per process is exhausted.
• The native heap is exhausted.
• The system has insufficient RAM and swap space.

Diagnostic Steps

• Verify -XX:+HeapDumpOnOutOfMemoryError is specified and OutOfMemoryError occured at the time in question
• If running on Red Hat Enterprise Linux (RHEL), see /var/log/messages to check if oom-killer was invoked and java process was killed.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.