Unaccounted memory usage when running Red Hat Enterprise Linux as a VMware guest

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • VMware Virtual Platform ESX server
  • VMware Memory Control Driver ( vmmemctl.ko or vmware_balloon.ko )

Issue

  • What is consuming all of the memory?

  • Unable to account for memory usage on a RHEL VMware guest :

      # free
              total      used        free     shared    buffers   cached
  Mem:       7927536    6071588    1855948       0      22428     209132
  -/+ buffers/cache:    5840028    2087508
  Swap:     12582896         92    12582804

  • System shows high memory usage even though applications are not allocating that much memory :

      # ps aux | tr -s ' ' | awk 'BEGIN{sum=0}{ sum+=$6 }END{print sum}'
  709340

Resolution

  • This is normal and expected behaviour, under the circumstances (see "Root Cause")

  • If this behaviour is not desired, the option of not using the memory ballooning features of the hypervisor should be considered, but this can be evaluated only at hypervisor level, together with the administrator/architect team of said hypervisor.

  • If this feature is currently in use, it can be assumed that the memory available on the hypervisor is overbooked, by design and architecture, and doing without memory ballooning is not, in practice, an immediate solution.

  • A unilateral choice to disable the ballooning feature on the guest OS is not advised, and is likely to have an impact on the overall stability of the entire hypervisor and the guests hosted by it (including hypervisor-wide memory exhaustion, with virtual machines running out of memory, killing processes or rebooting to recover, depending on OS and workload, and/or performance issues).

Root Cause

  • This behavior is caused by the VMware Memory Control Driver "vmmemctl.ko" or "vmware_balloon.ko" module.
filename:      /lib/modules/2.6.18-92.el5/misc/vmmemctl.ko  
author:        VMware, Inc.  
description:   VMware Memory Control Driver

OR

filename: /lib/modules/2.6.32-279.11.1.el6.x86_64/kernel/drivers/misc/vmware_balloon.ko
license: GPL
alias: vmware_vmmemctl
alias: dmi:*:svnVMware*:*
version: 1.2.1.1-k
description: VMware Memory Control (Balloon) Driver
author: VMware, Inc.
srcversion: 6186AB61A8235A390A06A80

  • In some situations, such as when the ESX server is under heavy memory load, the server will control the vmmemctl module to allocate and pin guest memory in "physical" memory.

  • Those "physical" memory regions can then be reclaimed by ESX server. This function is called "ballooning". Inflating the balloon increases memory pressure, and the kernel may page out memory pages to the swap space when memory pressure rises. The vmmemctl module does not have an external interface within the guest OS; it only communicates with the ESX server via a private channel. Therefore, the memory which is allocated by vmmemctl cannot be counted from the guest OS.

Diagnostic Steps

# cat /proc/meminfo
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB
  • Verify overall system memory usage from the output of free command.
    Eg:
# free
            total      used        free     shared    buffers   cached
Mem:       7927536    6071588    1855948       0      22428     209132
-/+ buffers/cache:    5840028    2087508
Swap:     12582896         92    12582804
  • Verify total memory utilization of process(es) running on system from the output of "ps" command output.
    Eg:
# ps aux | tr -s ' ' | awk 'BEGIN{sum=0}{ sum+=$6 }END{print sum}'
709340
  • Check system information from the output of dmidecode command.
    Eg:
# dmidecode | grep -A 2 "^System Information" 
System Information
	Manufacturer: VMware, Inc.
	Product Name: VMware Virtual Platform
  • Check VMware Memory Control Driver ( vmmemctl.ko or vmware_balloon.ko ) details.
# ps aux| grep -i vmmemctl 
root      2295  0.0  0.0      0     0 ?        S<   08:02   0:00 [vmmemctl]

# modinfo vmmemctl

or

# modinfo vmware_balloon
  • You can determine the usage of vmware_balloon from vmcore file by following the steps in the below:

How to determine the memory allocated to the VMware Balloon driver [vmw_balloon] (shipped with RHEL) from vmcore?

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.