How to identify JVM native memory segment allocations on OpenJDK / Oracle JDK

Solution Verified - Updated

Environment

  • Java application
  • Open JDK
  • Oracle JDK

Issue

  • We see a lot of [ anon ] segments when pmap is run on a Java process. How can we identify what they are?
  • How can we identify the cause of a JVM native memory leak?

Resolution

JVM options
Oracle JDK 7u40 and equivalent OpenJDK versions (2.4.0 and later) have some support for tracking native memory. See Content from hirt.se is not included.Content from hirt.se is not included.http://hirt.se/blog/?p=401 and <Content from docs.oracle.com is not included.https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html> for details. The late enough OpenJDK also has this feature.

NOTES:

  1. Enabling NMT will result in a 5-10 percent JVM performance drop and memory usage for NMT adds 2 machine words to all malloc memory as malloc header. NMT memory usage is also tracked by NMT. (Content from docs.oracle.com is not included.Source)
  2. NMT is very limited. It will show direct memory and Unsafe.AllocateMemory and Unsafe.ReallocateMemory usage under "Internal" (JDK8) or "Other" (JDK11); however, it will not show memory allocated by Java libraries, C/C++ libraries , or any application native methods. A heap dump when the Java process size is large is better for troubleshooting.

Requires addition of the following options 

 -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics

Where

  • -XX:+UnlockDiagnosticVMOptions enables the feature
  • -XX:NativeMemoryTracking args can be off|detail|summary

The last option, -XX:+PrintNMTStatistics, will print the statistics when the JVM process stops. Instead of this behaviour, you can execute the following command periodically.

jcmd $PID VM.native_memory detail > nmdetail.$(date --iso-8601=seconds)

Or you can use "baseline" and "detail.diff" features to calculate the differences. See jcmd $PID help VM.native_memory for all options.

jcmd $PID VM.native_memory baseline
jcmd $PID VM.native_memory detail.diff > nmdetail.$(date --iso-8601=seconds)

From OpenJDK11u11+ and OpenJDK17, jcmd <pid> VM.info has glibc malloc statistics from mallinfo2, introduced by Content from bugs.openjdk.org is not included.JDK-8251255. If this value continues to increase but the cause cannot be tracked by NMT, it is suspected that memory allocation due to malloc calls inside JNI methods.

Process Memory:
Virtual Size: 2743852K (peak: 4443332K)
Resident Set Size: 399496K (peak: 399496K) (anon: 371320K, file: 28176K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 95566K, retained: 74349K  // allocations = glibc_mallinfo2.uordblks + glibc_mallinfo2.hblkhd, retained = glibc_mallinfo2.fordblks
glibc malloc tunables: MALLOC_ARENA_MAX=1

Valgrind and GDB techniques will only work if you have debug symbols installed - which in practice means you need to use OpenJDK and not the Sun/Oracle JDK. The Oracle builds have all their debugging symbols stripped, so unless you can get a debug build from Oracle (usually requiring a support contract), these techniques cannot be used with their JDK.

To install debug symbols, run "debuginfo-install java-1.7.0-openjdk java-1.7.0-openjdk-devel".

Valgrind by default has a limit of 500 threads (including JVM and GC threads), so if there may be more than that you need a custom build which sets VG_N_THREADS to a higher value.

Valgrind's Massif tool
Using valgrind has a very large performance impact and will reduce performance by a factor of 5-50. As such it is only suitable for situations that can be reproduced in a test environment and not production.

  1. Install the "valgrind" package
  2. Install debug symbols for OpenJDK, as above
  3. Create a file called "java-massif.sh" containing the following, and use "chmod a+x java-massif.sh" on it to make it executable. If you point to a different JAVA_HOME or java installation, use that in the script instead.
 #!/bin/sh
 valgrind --trace-children=yes --tool=massif java $*
  1. Edit your start script to use 'java-massif.sh' instead of 'java' and add "-Djava.compiler=NONE" to your list of Java options (in run.conf/standalone.conf/domain.conf for JBoss)
  2. run your application, which should report something like the following on the command line at the start:
==24901== 
==24901== Massif, a heap profiler
==24901== Copyright (C) 2003-2013, and GNU GPL'd, by Nicholas Nethercote
==24901== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==24901== Command: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.5.0.1.fc20.x86_64/jre-abrt/bin/java
  1. Once you have reproduced the problem shut it down.
  2. You should see one or two "massif.out.$PID" files in the directory. Run "ms_print massif.out.$PID > heap.txt". If you had two files, once should be considerable larger and use that.

The resulting heap.txt file will detail how native memory is allocated, and can be analysed to determine the source of the memory usage.

gdb with openjdk debug package

  1. Install the "gdb" package

  2. Install debug symbols for OpenJDK, as above

  3. Create a GDB script file containing

    set breakpoint pending on
    set width 0
    set height 0
    set verbose off
    handle all nostop
    break anon_mmap
    commands 1
    backtrace
    continue
    end
    run

  4. Modify your run.sh script (for EAP 5) to change this (the first non-background one, or both instances):

 eval \"$JAVA\" $JAVA_OPTS \

to this:

 eval gdb -x /path/to/gdb-autobt.txt --args \"$JAVA\" $JAVA_OPTS \
  1. Start JBoss redirecting the console output to a file, e.g. "bin/run.sh > console.log"
  2. Run pmap again (pmap -x $PID)

The console output should contain section similar to this one

Breakpoint 1, os::reserve_memory (bytes=bytes@entry=51118080, requested_addr=requested_addr@entry=0x0, alignment_hint=alignment_hint@entry=4096) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:2896
2896	  return anon_mmap(requested_addr, bytes, (requested_addr != NULL));
#0  os::reserve_memory (bytes=bytes@entry=51118080, requested_addr=requested_addr@entry=0x0, alignment_hint=alignment_hint@entry=4096) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:2896
#1  0x00007ffff795c94d in initialize (executable=false, noaccess_prefix=0, requested_address=<optimized out>, large=false, alignment=<optimized out>, size=51118080, this=0x7ffff70f2910) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/runtime/virtualspace.cpp:334
#2  ReservedSpace::ReservedSpace (this=0x7ffff70f2910, size=<optimized out>, alignment=0, large=<optimized out>, requested_address=<optimized out>, noaccess_prefix=0) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/runtime/virtualspace.cpp:53
#3  0x00007ffff7813fd9 in ParMarkBitMap::initialize (this=this@entry=0x7ffff7dac9c0 <PSParallelCompact::_mark_bitmap>, covered_region=...) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/parMarkBitMap.cpp:61
#4  0x00007ffff7858037 in PSParallelCompact::initialize () at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/psParallelCompact.cpp:858
#5  0x00007ffff781c655 in ParallelScavengeHeap::initialize (this=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/gc_implementation/parallelScavenge/parallelScavengeHeap.cpp:252
#6  0x00007ffff7937ba5 in Universe::initialize_heap () at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/memory/universe.cpp:928
#7  0x00007ffff7937ebf in universe_init () at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/memory/universe.cpp:784
#8  0x00007ffff75d8685 in init_globals () at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/runtime/init.cpp:101
#9  0x00007ffff791e852 in Threads::create_vm (args=<optimized out>, canTryAgain=canTryAgain@entry=0x7ffff70f2e20) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/runtime/thread.cpp:3220
#10 0x00007ffff7641770 in JNI_CreateJavaVM (vm=0x7ffff70f2ea0, penv=0x7ffff70f2eb0, args=<optimized out>) at /usr/src/debug/java-1.7.0-openjdk-1.7.0.25-2.3.10.10.fc19.x86_64/openjdk/hotspot/src/share/vm/prims/jni.cpp:5127
#11 0x00007ffff7dcd9d8 in InitializeJVM (ifn=<synthetic pointer>, penv=0x7ffff70f2eb0, pvm=0x7ffff70f2ea0) at ../../../src/share/bin/java.c:1120
#12 JavaMain (_args=<optimized out>) at ../../../src/share/bin/java.c:358
#13 0x00000036f7407c53 in start_thread (arg=0x7ffff70f3700) at pthread_create.c:308
#14 0x00000036f70f513d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

By matching the 'bytes' address from that with the addresses in the pmap output, we will be able to determine what each of the segments are for.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.