kdump memory usage improvements included in Red Hat Enterprise Linux 6.2

Issue

This document describes some of the improvements that have been made in Red Hat Enterprise Linux (RHEL) 6.2 regarding kdump reserved memory usage and requirements.

Environment

• Red Hat Enterprise Linux 6.2 (x86/x86_64)

kdump memory usage improvements

One of the problems observed was the kdump kernel required more memory on machines with many cpus. This was confusing since the kdump kernel was booted with maxcpus=1 and bringing up only one cpu. However it was discovered that per cpu data memory was still being allocated for all of the cpus, and thus on larger systems more memory was required for kdump. Starting in RHEL 6.2 the kdump kernel is booted with nr_cpus=1 instead of maxcpus=1. This has led to very significant reduction in the memory required on larger systems.

It was also noticed that the amount of memory required was more on systems with large amounts of RAM. This is because the dump filtering utility (makedumpfile) uses 2bits of RAM per physical page of memory. This translates to 64MB of RAM used by the filtering utility per 1TB of physical RAM present in the system.

Command line syntax to reserve kdump memory

By default RHEL 6 is configured to use the crashkernel=auto kernel command line option to reserve memory for the kdump kernel. In RHEL 6.2 (x86/x86_64) the "auto" option has been fixed to properly scale reserved memory according to physical memory present in the system.

It is dificult to reliably predict the exact amount of memory needed for kdump kernel as it varies based on many factors. The updated crashkernel=auto logic uses the following calculation to determine the amount of memory to reserve:

reserved memory = 128MB + 64MB of memory per TB of physical RAM

So if a system has 1TB of memory crashkernel=auto will reserve 192MB (128MB + 64MB) of memory. This should ensure the amount of reserved memory is sufficent on wide variety of systems. However if there are (for example) many storage cards in the system and the associated drivers consume a lot of memory in the kdump kernel then additional memory may need to be reserved. The only way to ensure that enough memory has been reserved is to test kdump prior to placing the machine into production.

Note: Currently the crashkernel=auto option only reserves memory if the system has 4GB of RAM or more.

How to determine what is using the reserved memory

RHEL 6.2 adds some debugging aids which help to understand memory usage in the kdump kernel. This will help to determine if enough memory has been reserved, or if too much is reserved. It is strongly recommended to always test kdump after configuring a system to make sure the amount of memory reserved is sufficient.

The new debugging option debug_mem_level can be specified in /etc/kdump.conf. It has various levels (0-3) with varying verbosity of the output. When enabled it prints memory usage information to the console at various phases while booting the kdump kernel. For example, memory usage is printed after loading every module in the kdump kernel, before the dump copy is started and after dump copy is done. This allows a user to figure out what are the biggest consumers of memory.

As an example, with debug_mem_level 1 set in /etc/kdump.conf the following are some excerpts from the console output in the kdump kernel:

[debug_mem] At init start
MemTotal:         117064 kB
MemFree:          83616 kB
Cached:           11768 kB
Slab:             18292 kB

Loading dm-mod.ko module
[debug_mem] After module dm-mod.ko has been loaded
MemFree:           83032 kB
Cached:            11768 kB
Slab:              18764 kB

..

Loading ata_piix.ko module
[debug_mem] After module ata_piix.ko has been loaded
MemFree:           79640 kB
Cached:            11768 kB
Slab:              20856 kB

..

[debug_mem] Before dumping vmcore
MemFree:           40088 kB
Cached:            20544 kB
Slab:              48928 kB

Copying data                       : [100 %]
Saving core complete
[debug_mem] After dumping vmcore
MemFree:           4956 kB
Cached:            55876 kB
Slab:              50268 kB

Notice the free memory "Before dumping vmcore". At this point the kdump kernel is booted, all the drivers are loaded and we are about to start filtering and saving the vmcore. After this point the primary consumer of memory is going to be the filtering utility (2bits per physical page) as well as some memory required by the copying utilities and kernel caches.

The system in the above example has 44GB of RAM, so approxomately 3MB will be consumed by the filtering utility and approximately another 20MB is needed for the actual copying. So 23MB of free memory at this point should be enough.

If you have 1TB of RAM, 64MB will be used by the filtering utility and approxomately another 20MB for the actual copying. So 84MB of free memory at this point would be needed for a 1TB system.