How do I enable large page support on Linux?

Environment

• Red Hat Enterprise Linux
• Java JDK

Issue

• I want to enable large page support for Java, and I must enable it on my Linux system.

Resolution

• Verify the kernel supports large page memory:

$ cat /proc/meminfo | grep Huge
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

If the output shows the three "Huge" variables above, kernel supports large page memory.

Large page support is included in the 2.6 kernel. Some vendors have backported the code to their 2.4 based releases.

• Log in as root and set the shmmax, nr_hugepages, and hugetlb_shm_group (unless running as root, since the default value is 0 and matches the root group id) kernel parameters in /etc/sysctl.conf For example:

kernel.shmmax = 68719476736
vm.nr_hugepages = 1536
vm.hugetlb_shm_group = 500

The maximum shared memory (shmmax) is typically set to the amount of available physical memory.

For Java applications, the number of huge pages (nr_hugepages) is set to the total amount of memory the JVM will consume (heap, perm gen space, thread stacks, native code) divided by the meminfo Hugepagessize. For example, consider a Java application with a 2GB heap and 256MB perm gen space:

Total memory = (# of application instances) * (2GB (heap) + 256MB (perm) + 300-400MB (native JVM estimate) + (# of threads * threadstack size)). I will round up to 3GB to make the math easier:

nr_hugepages = (3)(1024)(1024)(1024)/(2048)(1024) = 1536

The group with permission to allocate large pages (hugetlb_shm_group) must be set to the group id (/etc/group) of the user running the JVM (unless running as root, since the default value is 0 and matches the root group id).

Run the following command to implement the changes in /etc/sysctl.conf:

# sysctl -p /etc/sysctl.conf

Verify the number of large pages was allocated:

# grep HugePages_Total /proc/meminfo

If HugePages_Total does not equal nr_hugepages, reboot and check again. Address space and allocated memory must be contiguous, and rebooting will resolve any address space or physical memory fragmentation.

• Log in as root and set the maximum locked-in-memory address space with the memlock directive in /etc/security/limits.conf for the user running the JVM. For example:

jboss     soft     memlock     3145728
jboss     hard     memlock     3145728</code>

The memlock value is expressed in Kilobytes and is the number of large pages (nr_hugepages) multiplied by the huge page size (Hugepagesize). For our 3GB example: memlock = (1536)(2048) = 3,145,728.

• The settings in limits.conf are not applied to  programs that are started as system services. If starting the Java application via init or as a system service use the ulimit command to manually set the limit in the shell environment that spawns the process the init script).
  

• Before starting the Java application, test to see if the ulimit changes have been picked up by running the following as the Java user:

# ulimit -a

If the ulimit settings were made after a shell was opened, it will be necessary to log back in to have the ulimit settings take affect:

# su - javauser

• Check the HugePages_Rsvd value in /proc/meminfo when the JVM is running. If large page support was successful, a non-zero value will be displayed. If the value is zero, double check that the ulimit settings are being picked up. Run as root to test if an issue is ulimit related (root is unlimited by default).
• Check standard out for errors. For example: Java application will not start due to "Failed to reserve shared memory" error on Linux
• Check /var/log/messages for "avc: denied" entries related to SELinux policies denying access to the shared memory segment.
• Finally, enable large page support for the JVM

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.