Enabling hugepages for use with Oracle Database

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • Oracle Database Application

Issue

  • Frequent unresponsiveness of server and application during heavy load
  • Near constant io wait on the server above 30%
  • How do I determine how many hugepages to preallocate for my Oracle instance?

Resolution

1. Calculate the amount of hugepages that will need to be allocated for your Oracle instance. 

NUMBER_OF_HUGEPAGES = (SGA size in bytes) / HugePage size

2. SGA is the value that is obtained from the Oracle instance. The correct number here is crucial. If you do not allocate enough hugepages for use with Oracle, zero pages will be used, causing "lost" memory from your available pool.

3. Enable the oracle user to be able to utilize huge pages in the /etc/sysctl.conf file.

vm.hugetlb_shm_group=`id -g oracle`

4. Set the amount of huge pages to allocate in the /etc/sysctl.conf file.

vm.nr_hugepages=NUMBER_OF_HUGEPAGES

Note: NUMBER_OF_HUGEPAGES should be replaced with the number calculated in step 1.

5. Calculate the amount of pages to put into the 'memlock' parameter.

AMOUNT_OF_MEMORY = NUMBER_OF_HUGEPAGES * 1024 * 2

6. Set the 'memlock' parameter in the /etc/security/limits.conf file.

oracle    -    memlock    AMOUNT_OF_MEMORY

Note: AMOUNT_OF_MEMORY should be replaced with the value calculated from step 5.

7. Restart the system.

  • These settings can be changed on the fly in a live environment; however, memory defragmentation will begin to occur which will have a negative impact on the system and could result in the system being unresponsive for an extended period of time while a large block of free memory is allocated. It is recommended to restart the machine completely to avoid this from happening.

  • If you're having trouble calculating your correct amount of huge pages, you may be able to use the following script to automatically determine the correct value

#!/bin/bash
 KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

 # Find out the HugePage size
  HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
            
 # Start from 1 pages to be on the safe side and guarantee 1 free HugePage
  NUM_PG=1
            
 # Cumulative number of pages required to handle the running shared memory segments
  for SEG_BYTES in `ipcs -m | awk '{print $5}' | grep "[0-9][0-9]*"`
   do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
      NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
  done
            
 # Finish with results
  case $KERN in
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.10')echo "Recommended setting: vm.nr_hugepages = $NUM_PG";;
        *) echo "Unrecognized kernel version $KERN. Exiting." ;;
  esac

Root Cause

  • By default, the kernel will attempt to make the most efficient use of your memory by swapping pages out from memory to disk, or by caching disk within the memory. This works well for almost all environments, however, when utilizing a database application, you may see problems occurring such as high iowait or an unresponsive server due to everything waiting on the storage.  This is caused by a high visit-to-disk count when trying to swap in and swap out pages of memory while your application is trying to access data.

  • Huge pages allows for a large pre-allocated amount of space be set aside for use by your application. This memory does not swap in/out from disk, and should alleviate some of the pressure that was previously seen in the environment.  It also provides other advantages, such as lower overhead for memory management.

  • HugePages can give substantial benefits in virtual memory management. Without HugePages, the memory of the SGA is divided into 4K pages, which have to be managed by the Linux kernel. Using HugePages, the page size is increased to 2MB , thereby reducing the total number of pages to be managed by the kernel and therefore reducing the amount of memory required to hold the page table in memory. The effort of page management make HugePages pretty much mandatory for Oracle systems running on x86-64 architectures.

Diagnostic Steps

  • If hugepages are not enabled a very high pagetables is seen on the host .
cat proc/meminfo 
MemTotal:       339797148 kB
MemFree:        27996108 kB
MemAvailable:   60743200 kB
Buffers:           73684 kB
Cached:         145427432 kB
SwapCached:        25904 kB
Active:         144943620 kB
Inactive:       20549436 kB
Active(anon):   127368844 kB
Inactive(anon):  6464768 kB
Active(file):   17574776 kB
Inactive(file): 14084668 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      104853500 kB
SwapFree:       104759604 kB
Dirty:               616 kB
Writeback:            28 kB
AnonPages:      19966252 kB
Mapped:         103242396 kB
Shmem:          113841668 kB
Slab:            3128804 kB
SReclaimable:    2681720 kB
SUnreclaim:       447084 kB
KernelStack:       39280 kB
PageTables:     138980872 kB <===========================
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.