How to find out what amount of memory a VMWare balloon driver has consumed from a virtualized server

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • The server is VMWare virtualized
  • Red Hat shipped balloon driver is used in the system

Issue

  • The system is running low on the memory, still there is no process or processes which are consuming the memory.

Resolution

Live system

  • If vmware-toolbox-cmd is present on the system, then during the runtime this can be found by running such command:

    vmware-toolbox-cmd stat balloon

which should output the amount of memory currently "ballooned". If such command is not present, please proceed with the steps below.

  • Mount debugfs filesystem by running:

    # mount -t debugfs nodev /sys/kernel/debug

  • Check /sys/kernel/debug/vmmemctl file:

    # cat /sys/kernel/debug/vmmemctl
target:              2059978 pages
current:             2059978 pages
rateNoSleepAlloc:      16384 pages/sec
rateSleepAlloc:         2048 pages/sec
rateFree:              16384 pages/sec

timer:                  1275
start:                     1 (   0 failed)
guestType:                 1 (   0 failed)
lock:                2060022 (  30 failed)
unlock:                   14 (   0 failed)
target:                 1275 (   0 failed)
primNoSleepAlloc:    2060022 (   0 failed)
primCanSleepAlloc:         0 (   0 failed)
primFree:                 14
errAlloc:                 30
errFree:                  30

  • current: line is showing the amount of memory in pages currently consumed by the balloon driver.

  • target: line is showing the amount of memory in pages the balloon driver is going to consume.

  • The above applies only to Red Hat shipped balloon driver for VMWare virtualized RHEL6 systems. RHEL5 systems are not shipped with Red Hat provided balloon driver, see VMware balloon driver support in RHEL article for the details. The above does not apply to VMWare shipped balloon driver, such an information can be obtained from the hypervisor side.

  • To calculate the amount of memory the balloon driver has consumed, just multiply the number of pages by the page size of your hardware architecture. For this given example, assuming this is a x86 architecture (which has page size of 4 KiB), the total memory used by the ballooning driver is 2059978 times 4 KiB, which gives approximately 7.86 GiB.

Sosreport from VMware virtual machine

  • Note that vmware plugin needs to be enabled for sos report. On Red Hat Enterprise Linux 8 and above, it is enabled on VMware virtual machines. Otherwise, it can be enabled manually by adding -e vmware to sosreport command line.

  • You will be able to extract balloon size (in KiB) by unpacking the sosreport and running the following command from sosreport's main directory:

    $ grep guest.mem.ballooned sos_commands/vmware/vmware-toolbox-cmd_stat_raw_text_resources
guest.mem.ballooned = 64333824

    In the example above, balloon size was 64333824 KiB, which translates to 61.35 GiB.

  • Please review below article by Broadcom how to disable Memory Ballooning on VM settings:

Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Root Cause

  • Some of the "physical" memory on the virtual guest may be held by so-called balloon driver. This kernel module is "vmmemctl" or "vmware_balloon" or "vmw_balloon". The following commands can be used to confirm the module is loaded into the system:

    lsmod | grep vmmemctl
lsmod | grep vmware_balloon
lsmod | grep vmw_balloon

  • In the situation of heavy memory load happened on the hypervisor (VMWare ESX/ESXi server), the hypervisor will instruct this module to allocate guest memory pages and pin them in the "physical" guest memory. Those "physical" memory can then be reclaimed by the hypervisor. This function is called "ballooning". Inflating the balloon increases the memory pressure in the guest, the guest operating system may page out memory pages to the swap space when its memory is scarce.

  • VMWare shipped ballooning module do not have an external interface within the guest OS, it only communicates with the hypervisor via a private channel. Therefore, the memory which allocated by this module cannot be counted from guest OS. Please, see more details in Unaccounted memory usage when running Red Hat Enterprise Linux as a VMware guest article. For more information about VMWare shipped balloon driver, please refer to the documentation on Vmware website or contact Vmware support.

  • Red Hat shipped balloon driver "vmware_balloon" in RHEL6 has an interface within the guest OS through "debugfs" filesystem and can be used as described in the Resolution section.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.